cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Somi
by New Contributor III
  • 4274 Views
  • 10 replies
  • 0 kudos

Resolved! How to set sparkTrials? I am receiving this TypeError: cannot pickle '_thread.lock' object

I am trying to distribute hyperparameter tuning using hyperopt on a tensorflow.keras model. I am using sparkTrials in my fmin:spark_trials = SparkTrials(parallelism=4)...best_hyperparam = fmin(fn=CNN_HOF,                  space=space,                ...

  • 4274 Views
  • 10 replies
  • 0 kudos
Latest Reply
Dooley
Valued Contributor
  • 0 kudos

This can happen when you try to serialize a keras model with an unserializable layer. What does your model look like? Also what is in that search space variable? What are you trying to optimize on?

  • 0 kudos
9 More Replies
Joseph_B
by New Contributor III
  • 955 Views
  • 2 replies
  • 0 kudos

For tuning hyperparameters with Apache Spark ML / MLlib, when should I use Spark ML's built-in tuning algorithms vs. Hyperopt?

When should I use Spark ML's CrossValidator or TrainValidationSplit, vs. a separate tuning tool such as Hyperopt?

  • 955 Views
  • 2 replies
  • 0 kudos
Latest Reply
Joseph_B
New Contributor III
  • 0 kudos

Both are valid choices. By default, I'd recommend using Hyperopt nowadays. Here's the rationale, as pros & cons of each.Spark ML's built-in toolsPros: These fit the Spark ML Pipeline framework, so you can keep using the same type of APIs.Cons: Thes...

  • 0 kudos
1 More Replies
User16789201666
by Contributor II
  • 548 Views
  • 0 replies
  • 0 kudos

What's a best practice for Hyperopt workflow?

Choose what hyperparameters are reasonable to optimizeDefine broad ranges for each of the hyperparameters (including the default where applicable)Run a small number of trialsObserve the results in an MLflow parallel coordinate plot and select the run...

  • 548 Views
  • 0 replies
  • 0 kudos
User16789201666
by Contributor II
  • 2415 Views
  • 0 replies
  • 0 kudos

When to use uniform vs log-uniform in Hyperopt?

Hyperopt offers hp.uniform and hp.loguniform, both of which produce real values in a min/max range. hp.loguniform is more suitable when one might choose a geometric series of values to try (0.001, 0.01, 0.1) rather than arithmetic (0.1, 0.2, 0.3). Wh...

  • 2415 Views
  • 0 replies
  • 0 kudos
Labels