Databricks Community

Somi · 08-23-2022

Hey Sara, this Somayeh from VINN Automotive.As I had already shared with you, I am trying to distribute hyperparameter tuning using hyperopt on a tensorflow.keras model. I am using sparkTrials in my fmin:spark_trials = SparkTrials(parallelism=4)...be...

Somi · 08-23-2022

I am trying to distribute hyperparameter tuning using hyperopt on a tensorflow.keras model. I am using sparkTrials in my fmin:spark_trials = SparkTrials(parallelism=4)...best_hyperparam = fmin(fn=CNN_HOF, space=space, ...

Somi · 06-24-2022

I have saved a keras model in some directories in dbfs to load and retrain that with more data, etc. The problem is that when cluster stops and restarts, seems those directories and model are no longer available there and it starts training a new mod...

Somi · 06-24-2022

I am trying to fit a model with callbacks including Tensorboard, history and checkpoint.Then I am loading the model and when trying to fit it again on more epochs, I am receiving this error:UnimplementedError: /dbfs/<path_to_history dir>/history.csv;...

Somi · 09-02-2022

Yes, all those are global variables. This code was not working for me, but it gave me the clue. I changed spark dataframes to pandas outside of the image generators and then I moved Image generators inside of the objective function. This way there wa...

Somi · 09-02-2022

Hi @Vidula Khanna I figured it out by replacing OS library module with dbutils utilities. It looks like mre compatible with DBFS.

Somi · 08-31-2022

Using sparktrials I am receiving this error not the same error I was receiving before:BadObjectiveFunction: When using `fmin` asynchronously, distributed algorithms or distributed objects may not be used within the objective function. This includes a...

Somi · 08-30-2022

I changed the code in the way you were suggesting. It was not working with sparktrials(). It only worked with Trials() which I believe means there is no distributed tuning as we don't call distributed training algorithms such as MLlib or Horovod and ...

Somi · 08-30-2022

This is a summary of the cluster:2-8 Workers 32-128 GB Memory 8-32 Cores1 Driver 16 GB Memory, 4 CoresRuntime 10.5.x-gpu-ml-scala2.12 (3) what is the value set for tuner_max_evals? Right now it's been set to 36, but with every value o...

Databricks Community

User Stats

User Activity

How to set sparkTrials? I am receiving this TypeError: cannot pickle '_thread.lock' object

How to set sparkTrials? I am receiving this TypeError: cannot pickle '_thread.lock' object

No saved model after stopping the cluster.

UnimplementedError; Operation not supported

Re: How to set sparkTrials? I am receiving this TypeError: cannot pickle '_thread.lock' object

Re: No saved model after stopping the cluster.

Re: How to set sparkTrials? I am receiving this TypeError: cannot pickle '_thread.lock' object

Re: How to set sparkTrials? I am receiving this TypeError: cannot pickle '_thread.lock' object

Re: How to set sparkTrials? I am receiving this TypeError: cannot pickle '_thread.lock' object