Hi, aromano

I know this issue was opened almost a year ago, but I faced the same problem and I was able to solve it. So, I'm sharing the solution in order to help others.

Probably, you're using SparkTrials to optimize the model's hyperparameters in Databricks, In this case, you need to do 3 things:

1.define two environment variables:

import os
os.environ["DATABRICKS_HOST"] = "<YOUR DATABRICKS HOST>"
os.environ["DATABRICKS_TOKEN"] = "<YOUR DATABRICKS TOKEN>"

2. register spark as a backend for joblib.Parallel:

from joblibspark import register_spark
register_spark()

3. define the joblib.Parallel as "spark". For instance:

Parallel(n_jobs=-1, backend="spark")

I hope it helps