topic Re: Distributed Optuna and MLflow in Machine Learning

Distributed Optuna and MLflow

Edwin1 — Sat, 06 Sep 2025 17:30:18 GMT

Hello All,

I just tried running the following notebook (https://docs.databricks.com/aws/en/notebooks/source/machine-learning/optuna-mlflow.html) on the Databricks Free Edition platform , through Microsoft Account Authentication. It takes 15 minutes to run 8 trials then outputs a grpc_status:13 error before it breaks. Could this be because the distributed MlflowSparkStudy class cannot work on the Free Edition, or is this an internal bug.

Thank you in advance !

Re: Distributed Optuna and MLflow

Edwin1 — Sat, 06 Sep 2025 17:47:07 GMT

For what it's worth, I just realized the free edition is a single node cluster, so this is mute point anyway.

Thank you!

Re: Distributed Optuna and MLflow

BS_THE_ANALYST — Sat, 06 Sep 2025 17:52:52 GMT

@Edwin1
I can appreciate this isn't a solution with proper reasoning but changing the following in your notebook, on the free edition, should allow you to use the notebook:
I reduced the number of trials and it worked 🤔.

Best guess it's around the compute limitations? Not a definitive conclusion though.

Here's AI's take on the situation:

Hopefully that's a step in the right direction for you @Edwin1 ☺️.

All the best,
BS

Re: Distributed Optuna and MLflow

Edwin1 — Sat, 06 Sep 2025 18:22:40 GMT

Great. Thank you. That worked. I still need more compute and networking resources to make it justifiable, but this confirms that it works !!!

Re: Distributed Optuna and MLflow

BS_THE_ANALYST — Sun, 07 Sep 2025 09:54:33 GMT

No worries ☺️. If you end up building out a cool project with the Free Edition, I'm sure the community would love to hear about it. I've not got around to anything ML-related in Databricks yet, very eager though 🫡. There's a dedicated space for it https://community.databricks.com/t5/community-articles/bd-p/Knowledge-Sharing-Hub

Best of luck with the project.

All the best,
BS