What you are encountering is an expected behavior on Databricks serverless. Because CrossValidator (along with other Spark ML tuning estimators) relies heavily on internal DataFrame caching to optimize iterative model training, Databricks enforces strict security and data isolation boundaries on it. This underlying architecture dictates that any temporary storage utilized for caching must be explicitly routed to a Unity Catalog (UC) Volume, rather than relying on local, ephemeral cluster storage.
You can set the SPARKML_TEMP_DFS_PATH environment variable at the beginning of the notebook, strictly prior to instantiating your CrossValidator. Because notebook-level variables do not persist across cluster restarts, this must be executed at the start of every session.
import os
# Create a UC volume for Spark ML temporary storage
spark.sql("CREATE VOLUME IF NOT EXISTS main.default.spark_ml_temp")
# Set the environment variable (required in every session)
os.environ["SPARKML_TEMP_DFS_PATH"] = "/Volumes/main/default/spark_ml_temp"
You will need to create a standard UC Volume specifically designated for temporary ML storage. You can use any valid UC volume path, provided the executing user or service principal has the requisite WRITE VOLUME privileges on the path.
Be aware that this internal caching behavior is not exclusive to CrossValidator. If you are utilizing TrainValidationSplit or other native Spark ML estimators that trigger DataFrame caching, you will need to implement this exact same volume mapping to avoid runtime failures.
