Distributed Optuna and MLflow

[Edwin1]

Hello All, 

I just tried running the following notebook (https://docs.databricks.com/aws/en/notebooks/source/machine-learning/optuna-mlflow.html)  on the Databricks Free Edition platform , through Microsoft Account Authentication. It takes 15 minutes to run 8 trials then outputs a grpc_status:13 error before it breaks. Could this be because the distributed MlflowSparkStudy class cannot work on  the Free Edition, or is this an internal bug.

Thank you in advance !

[Edwin1]

For what it's worth, I just realized the free edition is a single node cluster, so this is mute point anyway.

Thank you!

[BS_THE_ANALYST · Accepted solution]

@Edwin1 
I can appreciate this isn't a solution with proper reasoning but changing the following in your notebook, on the free edition, should allow you to use the notebook:
 I reduced the number of trials and it worked 🤔.

Best guess it's around the compute limitations? Not a definitive conclusion though. 

Here's AI's take on the situation:

Hopefully that's a step in the right direction for you @Edwin1  ☺️.

All the best,
BS

[Edwin1]

Great. Thank you. That worked. I still need more compute and networking resources to make it justifiable, but this confirms that it works !!!

[BS_THE_ANALYST]

No worries ☺️. If you end up building out a cool project with the Free Edition, I'm sure the community would love to hear about it. I've not got around to anything ML-related in Databricks yet, very eager though 🫡. There's a dedicated space for it https://community.databricks.com/t5/community-articles/bd-p/Knowledge-Sharing-Hub 

Best of luck with the project.

All the best,
BS