- 5983 Views
- 10 replies
- 0 kudos
Resolved! How to set sparkTrials? I am receiving this TypeError: cannot pickle '_thread.lock' object
I am trying to distribute hyperparameter tuning using hyperopt on a tensorflow.keras model. I am using sparkTrials in my fmin:spark_trials = SparkTrials(parallelism=4)...best_hyperparam = fmin(fn=CNN_HOF, space=space, ...
- 5983 Views
- 10 replies
- 0 kudos
- 0 kudos
This can happen when you try to serialize a keras model with an unserializable layer. What does your model look like? Also what is in that search space variable? What are you trying to optimize on?
- 0 kudos
- 1415 Views
- 2 replies
- 0 kudos
For tuning hyperparameters with Apache Spark ML / MLlib, when should I use Spark ML's built-in tuning algorithms vs. Hyperopt?
When should I use Spark ML's CrossValidator or TrainValidationSplit, vs. a separate tuning tool such as Hyperopt?
- 1415 Views
- 2 replies
- 0 kudos
- 0 kudos
Both are valid choices. By default, I'd recommend using Hyperopt nowadays. Here's the rationale, as pros & cons of each.Spark ML's built-in toolsPros: These fit the Spark ML Pipeline framework, so you can keep using the same type of APIs.Cons: Thes...
- 0 kudos
- 798 Views
- 0 replies
- 0 kudos
What's a best practice for Hyperopt workflow?
Choose what hyperparameters are reasonable to optimizeDefine broad ranges for each of the hyperparameters (including the default where applicable)Run a small number of trialsObserve the results in an MLflow parallel coordinate plot and select the run...
- 798 Views
- 0 replies
- 0 kudos
- 3200 Views
- 0 replies
- 0 kudos
When to use uniform vs log-uniform in Hyperopt?
Hyperopt offers hp.uniform and hp.loguniform, both of which produce real values in a min/max range. hp.loguniform is more suitable when one might choose a geometric series of values to try (0.001, 0.01, 0.1) rather than arithmetic (0.1, 0.2, 0.3). Wh...
- 3200 Views
- 0 replies
- 0 kudos