Free Edition CrossValidator not working because of internal caching

pjvi
New Contributor II

Hi all,

I am getting the following error when using CrossValidator in Databricks Free Edition v5 (see attachment):

In shared or serverless cluster, SPARK_ML_TMP_DFS_PATH environmental variable must be set to a UC volume path like '/Volumes/...' in order to support Saprk DataFrame caching

Could you please check if this the expected behaviour, and if there a workaround or something we can try?

Thanks