cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Machine Learning
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Somi
by • New Contributor III
  • 4006 Views
  • 10 replies
  • 0 kudos

Resolved! How to set sparkTrials? I am receiving this TypeError: cannot pickle '_thread.lock' object

I am trying to distribute hyperparameter tuning using hyperopt on a tensorflow.keras model. I am using sparkTrials in my fmin:spark_trials = SparkTrials(parallelism=4)...best_hyperparam = fmin(fn=CNN_HOF,                  space=space,                ...

Machine Learning
Reply
  • 4006 Views
  • 10 replies
  • 0 kudos
View Replies
Latest Reply
Dooley
Valued Contributor
  • 0 kudos

This can happen when you try to serialize a keras model with an unserializable layer. What does your model look like? Also what is in that search space variable? What are you trying to optimize on?

  • 0 kudos
9 More Replies
Joseph_B
by • New Contributor III
  • 887 Views
  • 2 replies
  • 0 kudos

For tuning hyperparameters with Apache Spark ML / MLlib, when should I use Spark ML's built-in tuning algorithms vs. Hyperopt?

When should I use Spark ML's CrossValidator or TrainValidationSplit, vs. a separate tuning tool such as Hyperopt?

Machine Learning
Reply
  • 887 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Joseph_B
New Contributor III
  • 0 kudos

Both are valid choices. By default, I'd recommend using Hyperopt nowadays. Here's the rationale, as pros & cons of each.Spark ML's built-in toolsPros: These fit the Spark ML Pipeline framework, so you can keep using the same type of APIs.Cons: Thes...

  • 0 kudos
1 More Replies
Labels