What's the best way to use hyperopt to train a spark.ml model and track automatically with mlflow?

User16752240150 — Fri, 04 Jun 2021 19:34:03 GMT

I've read this article, which covers:

Using CrossValidator or TrainValidationSplit to track hyperparameter tuning (no hyperopt). Only random/grid search
parallel "single-machine" model training with hyperopt using hyperopt.SparkTrials (not spark.ml)
"Distributed training with Hyperopt and HorovodRunner" - distributed deep learning with hyperopt (no MLFlow)
- It does mention "With HorovodRunner, you do not use the SparkTrials class, and you must manually call MLflow to log trials for Hyperopt."

Is there an example notebook that shows how to hyperparameter tune a spark.ml model and log hyperparams/metrics/artifacts?

Re: What's the best way to use hyperopt to train a spark.ml model and track automatically with mlflow?

sean_owen — Fri, 18 Jun 2021 00:00:45 GMT

It's actually pretty simple: use hyperopt, but use "Trials" not "SparkTrials". You get parallelism from Spark, not from the tuning process.