- 6563 Views
- 10 replies
- 0 kudos
Resolved! How to set sparkTrials? I am receiving this TypeError: cannot pickle '_thread.lock' object
I am trying to distribute hyperparameter tuning using hyperopt on a tensorflow.keras model. I am using sparkTrials in my fmin:spark_trials = SparkTrials(parallelism=4)...best_hyperparam = fmin(fn=CNN_HOF, space=space, ...
- 6563 Views
- 10 replies
- 0 kudos
- 0 kudos
This can happen when you try to serialize a keras model with an unserializable layer. What does your model look like? Also what is in that search space variable? What are you trying to optimize on?
- 0 kudos
- 1530 Views
- 1 replies
- 0 kudos
For tuning hyperparameters with Apache Spark ML / MLlib, when should I use Spark ML's built-in tuning algorithms vs. Hyperopt?
When should I use Spark ML's CrossValidator or TrainValidationSplit, vs. a separate tuning tool such as Hyperopt?
- 1530 Views
- 1 replies
- 0 kudos
- 0 kudos
Both are valid choices. By default, I'd recommend using Hyperopt nowadays. Here's the rationale, as pros & cons of each.Spark ML's built-in toolsPros: These fit the Spark ML Pipeline framework, so you can keep using the same type of APIs.Cons: Thes...
- 0 kudos