Unable to use GridsearchCV from spark-sklearn due to 'fit_params' error

[gary7135]

When using GridsearchCV from spark-sklearn, I got GridSearchCV giving " __init__() got an unexpected keyword argument 'fit_params' error

I am using sklearn 1.2.2 and spark-sklearn 0.3.0

I think this is because spark-sklearn GridsearchCV still has the fit_params parameter which is deprecated in GridsearchCV in sklearn 1.2.2.

I am trying to use Databrick's parallel processing to help me tune hyper parameter using this example: https://kb.databricks.com/en_US/machine-learning/kfold-cross-validation

What would be a good solution ?

[Anonymous]

@Gary Mu​ :

Yes, you are correct. The error message you are seeing is likely due to the fact that the fit_params parameter was deprecated in GridSearchCV in sklearn 1.2.2.

One possible solution is to use a different version of scikit-learn that is compatible with spark-sklearn 0.3.0 and still has the fit_params parameter. You can try downgrading scikit-learn to a version prior to 1.2.2, such as 1.1.0, which should be compatible with spark-sklearn 0.3.0 and still have the fit_params parameter.

Alternatively, you could modify the GridSearchCV implementation in spark-sklearn to remove the fit_params parameter. You can do this by forking the spark-sklearn repository, modifying the code, and building a new version of the package. Another option is to use Hyperopt or Optuna for hyperparameter tuning instead of GridSearchCV . These libraries are designed for distributed computing and can be used with Databricks.