cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

RiyazAli
by Valued Contributor
  • 1660 Views
  • 2 replies
  • 3 kudos

Errors in notebooks of Scalable Machine Learning with Apache Spark course in Databricks academy.

HI there,I'm following the course mentioned from Databricks Academy. I downloaded the .dbc archiive and working along side the videos from academy. In ML-08 - Hyperopt notebook, I see the following error in cmd 13. best_hyperparam = fmin(fn=objectiv...

hyperopt_implementation hyperopt problem with "max_features"
  • 1660 Views
  • 2 replies
  • 3 kudos
Latest Reply
RiyazAli
Valued Contributor
  • 3 kudos

Tagging @Kaniz Fatma​ as there was no response what so ever!By any chance, do you know how to resolve these errors in the notebook?Thanks!

  • 3 kudos
1 More Replies
Joseph_B
by New Contributor III
  • 1198 Views
  • 2 replies
  • 0 kudos

How should I tune hyperparameters when fitting models for every item?

My dataset has an "item" column which groups the rows into many groups. (Think of these groups as items in a store.) I want to fit 1 ML model per group. Should I tune hyperparameters for each group separately? Or should I tune them for the entire...

  • 1198 Views
  • 2 replies
  • 0 kudos
Latest Reply
Joseph_B
New Contributor III
  • 0 kudos

For the first question ("which option is better?"), you need to answer that via your understanding of the problem domain.Do you expect similar behavior across the groups (items)?If so, that's a +1 in favor of sharing hyperparameters. And vice versa....

  • 0 kudos
1 More Replies
User16826994223
by Honored Contributor III
  • 447 Views
  • 0 replies
  • 0 kudos

Best practices: Hyperparameter tuning with Hyperopt Bayesian approaches can be much more efficient than grid search and random search. Hence, with the...

Best practices: Hyperparameter tuning with HyperoptBayesian approaches can be much more efficient than grid search and random search. Hence, with the Hyperopt Tree of Parzen Estimators (TPE) algorithm, you can explore more hyperparameters and larger ...

  • 447 Views
  • 0 replies
  • 0 kudos
User16752240150
by New Contributor II
  • 802 Views
  • 1 replies
  • 0 kudos

What's the best way to use hyperopt to train a spark.ml model and track automatically with mlflow?

I've read this article, which covers:Using CrossValidator or TrainValidationSplit to track hyperparameter tuning (no hyperopt). Only random/grid searchparallel "single-machine" model training with hyperopt using hyperopt.SparkTrials (not spark.ml)"Di...

  • 802 Views
  • 1 replies
  • 0 kudos
Latest Reply
sean_owen
Honored Contributor II
  • 0 kudos

It's actually pretty simple: use hyperopt, but use "Trials" not "SparkTrials". You get parallelism from Spark, not from the tuning process.

  • 0 kudos
Joseph_B
by New Contributor III
  • 523 Views
  • 1 replies
  • 0 kudos

When doing hyperparameter tuning with Hyperopt, when should I use SparkTrials? Does it work with both single-machine ML (like sklearn) and distributed ML (like Apache Spark ML)?

I want to know how to use Hyperopt in different situations:Tuning a single-machine algorithm from scikit-learn or single-node TensorFlowTuning a distributed algorithm from Spark ML or distributed TensorFlow / Horovod

  • 523 Views
  • 1 replies
  • 0 kudos
Latest Reply
Joseph_B
New Contributor III
  • 0 kudos

The right question to ask is indeed: Is the algorithm you want to tune single-machine or distributed?If it's a single-machine algorithm like any from scikit-learn, then you can use SparkTrials with Hyperopt to distribute hyperparameter tuning.If it's...

  • 0 kudos
Labels