cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

What's the best way to use hyperopt to train a spark.ml model and track automatically with mlflow?

User16752240150
New Contributor II

I've read this article, which covers:

  • Using CrossValidator or TrainValidationSplit to track hyperparameter tuning (no hyperopt). Only random/grid search
  • parallel "single-machine" model training with hyperopt using hyperopt.SparkTrials (not spark.ml)
  • "Distributed training with Hyperopt and HorovodRunner" - distributed deep learning with hyperopt (no MLFlow)
    • It does mention "With HorovodRunner, you do not use the SparkTrials class, and you must manually call MLflow to log trials for Hyperopt."

Is there an example notebook that shows how to hyperparameter tune a spark.ml model and log hyperparams/metrics/artifacts?

1 REPLY 1

sean_owen
Honored Contributor II
Honored Contributor II

It's actually pretty simple: use hyperopt, but use "Trials" not "SparkTrials". You get parallelism from Spark, not from the tuning process.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!