Free Edition CrossValidator not working because of internal caching

pjvi — Tue, 07 Jul 2026 21:05:51 GMT

Hi all,

I am getting the following error when using CrossValidator in Databricks Free Edition v5 (see attachment):

In shared or serverless cluster, SPARK_ML_TMP_DFS_PATH environmental variable must be set to a UC volume path like '/Volumes/...' in order to support Saprk DataFrame caching

Could you please check if this the expected behaviour, and if there a workaround or something we can try?

Thanks

Re: Free Edition CrossValidator not working because of internal caching

balajij8 — Wed, 08 Jul 2026 03:01:34 GMT

What you are encountering is an expected behavior on Databricks serverless. Because CrossValidator (along with other Spark ML tuning estimators) relies heavily on internal DataFrame caching to optimize iterative model training, Databricks enforces strict security and data isolation boundaries on it. This underlying architecture dictates that any temporary storage utilized for caching must be explicitly routed to a Unity Catalog (UC) Volume, rather than relying on local, ephemeral cluster storage.

You can set the SPARKML_TEMP_DFS_PATH environment variable at the beginning of the notebook, strictly prior to instantiating your CrossValidator. Because notebook-level variables do not persist across cluster restarts, this must be executed at the start of every session.

import os # Create a UC volume for Spark ML temporary storage spark.sql("CREATE VOLUME IF NOT EXISTS main.default.spark_ml_temp") # Set the environment variable (required in every session) os.environ["SPARKML_TEMP_DFS_PATH"] = "/Volumes/main/default/spark_ml_temp"

You will need to create a standard UC Volume specifically designated for temporary ML storage. You can use any valid UC volume path, provided the executing user or service principal has the requisite WRITE VOLUME privileges on the path.

Be aware that this internal caching behavior is not exclusive to CrossValidator. If you are utilizing TrainValidationSplit or other native Spark ML estimators that trigger DataFrame caching, you will need to implement this exact same volume mapping to avoid runtime failures.

topic Re: Free Edition CrossValidator not working because of internal caching in Machine Learning

Free Edition CrossValidator not working because of internal caching

Re: Free Edition CrossValidator not working because of internal caching