https://community.databricks.com/t5/data-engineering/how-to-set-sparktrials-i-am-receiving-this-typeerror-cannot/m-p/33723#M24661 <P>Hey Sara, this Somayeh from VINN Automotive.</P><P>As I had already shared with you, I am trying to distribute hyperparameter tuning using hyperopt on a tensorflow.keras model. I am using sparkTrials in my fmin:</P><P>spark_trials = SparkTrials(parallelism=4)</P><P>...</P><P>best_hyperparam = fmin(fn=CNN_HOF,&nbsp;</P><P>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;space=space,&nbsp;</P><P>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;algo=tpe.suggest,&nbsp;</P><P>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;max_evals=tuner_max_evals,</P><P>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;trials=spark_trials)</P><P>but I am receiving this error:</P><P>TypeError: cannot pickle '_thread.lock' object</P><P></P><P></P><P>the only way the code is working is skipping the trials passing by commenting out the line trials=spark_trials which means there would be no distributed tuning.</P><P></P><P>I went through the resources you shared, but couldn't find anything working for me though. Any idea how can I fix this?</P> Tue, 23 Aug 2022 17:47:47 GMT Somi 2022-08-23T17:47:47Z https://community.databricks.com/t5/data-engineering/how-to-set-sparktrials-i-am-receiving-this-typeerror-cannot/m-p/33723#M24661 <P>Hey Sara, this Somayeh from VINN Automotive.</P><P>As I had already shared with you, I am trying to distribute hyperparameter tuning using hyperopt on a tensorflow.keras model. I am using sparkTrials in my fmin:</P><P>spark_trials = SparkTrials(parallelism=4)</P><P>...</P><P>best_hyperparam = fmin(fn=CNN_HOF,&nbsp;</P><P>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;space=space,&nbsp;</P><P>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;algo=tpe.suggest,&nbsp;</P><P>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;max_evals=tuner_max_evals,</P><P>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;trials=spark_trials)</P><P>but I am receiving this error:</P><P>TypeError: cannot pickle '_thread.lock' object</P><P></P><P></P><P>the only way the code is working is skipping the trials passing by commenting out the line trials=spark_trials which means there would be no distributed tuning.</P><P></P><P>I went through the resources you shared, but couldn't find anything working for me though. Any idea how can I fix this?</P> Tue, 23 Aug 2022 17:47:47 GMT https://community.databricks.com/t5/data-engineering/how-to-set-sparktrials-i-am-receiving-this-typeerror-cannot/m-p/33723#M24661 Somi 2022-08-23T17:47:47Z