https://community.databricks.com/t5/machine-learning/tuning-with-optuna-and-mlflowsparkstudy/m-p/167830#M4698 <P>I am following the guide for tuning a model with Optuna and MlflowSparkStudy. My compute is configured with autoscaling enabled, with 1–2 Spark workers, each with 8 cores and 32 GB of memory. I set n_jobs=2 and trials=100, in <STRONG>mlflow_study.optimize()</STRONG>.</P><P>I have a few questions about how MlflowSparkStudy distributes the workload:</P><OL><LI><P><STRONG>How should I think about n_jobs in mlflow_study.optimize() vs. num_threads/n_jobs in LightGBM?</STRONG></P><P>For example, with 1–2 Spark workers and 8 cores per worker, would it make sense to set mlflow_study.optimize(n_jobs=2) and LightGBM n_jobs=7? My dataset is large, so ideally I would like to run only <STRONG>one model-training job per Spark worker</STRONG> and use the remaining cores on that worker for the LightGBM training.</P></LI><LI><P><STRONG>Does MlflowSparkStudy automatically trigger or make use of Spark autoscaling?</STRONG></P><P>If I configure my cluster with a minimum of 1 worker and a maximum of 2 workers, will MlflowSparkStudy cause Spark to scale up to 2 workers as needed when running multiple trials in parallel?</P></LI><LI><P><STRONG>What is the recommended way to make a large DataFrame available to each Spark worker?</STRONG></P><P>Currently, I have a Pandas DataFrame that I pass to the objective function used by mlflow_study.optimize(). Should I broadcast the DataFrame, cache it in Spark, or use another approach to avoid repeatedly transferring the data to each worker for every trial?</P></LI></OL><P>Any guidance on the recommended configuration or best practices would be greatly appreciated.</P><P>#mlflow #optuna #<SPAN>MlflowSparkStudy</SPAN></P><P><A href="https://learn.microsoft.com/en-us/azure/databricks/machine-learning/automl-hyperparam-tuning/optuna" target="_self">https://learn.microsoft.com/en-us/azure/databricks/machine-learning/automl-hyperparam-tuning/optuna</A>&nbsp;</P> Mon, 07 Sep 2026 19:24:05 GMT AdamIH123 2026-09-07T19:24:05Z https://community.databricks.com/t5/machine-learning/tuning-with-optuna-and-mlflowsparkstudy/m-p/167830#M4698 <P>I am following the guide for tuning a model with Optuna and MlflowSparkStudy. My compute is configured with autoscaling enabled, with 1–2 Spark workers, each with 8 cores and 32 GB of memory. I set n_jobs=2 and trials=100, in <STRONG>mlflow_study.optimize()</STRONG>.</P><P>I have a few questions about how MlflowSparkStudy distributes the workload:</P><OL><LI><P><STRONG>How should I think about n_jobs in mlflow_study.optimize() vs. num_threads/n_jobs in LightGBM?</STRONG></P><P>For example, with 1–2 Spark workers and 8 cores per worker, would it make sense to set mlflow_study.optimize(n_jobs=2) and LightGBM n_jobs=7? My dataset is large, so ideally I would like to run only <STRONG>one model-training job per Spark worker</STRONG> and use the remaining cores on that worker for the LightGBM training.</P></LI><LI><P><STRONG>Does MlflowSparkStudy automatically trigger or make use of Spark autoscaling?</STRONG></P><P>If I configure my cluster with a minimum of 1 worker and a maximum of 2 workers, will MlflowSparkStudy cause Spark to scale up to 2 workers as needed when running multiple trials in parallel?</P></LI><LI><P><STRONG>What is the recommended way to make a large DataFrame available to each Spark worker?</STRONG></P><P>Currently, I have a Pandas DataFrame that I pass to the objective function used by mlflow_study.optimize(). Should I broadcast the DataFrame, cache it in Spark, or use another approach to avoid repeatedly transferring the data to each worker for every trial?</P></LI></OL><P>Any guidance on the recommended configuration or best practices would be greatly appreciated.</P><P>#mlflow #optuna #<SPAN>MlflowSparkStudy</SPAN></P><P><A href="https://learn.microsoft.com/en-us/azure/databricks/machine-learning/automl-hyperparam-tuning/optuna" target="_self">https://learn.microsoft.com/en-us/azure/databricks/machine-learning/automl-hyperparam-tuning/optuna</A>&nbsp;</P> Mon, 07 Sep 2026 19:24:05 GMT https://community.databricks.com/t5/machine-learning/tuning-with-optuna-and-mlflowsparkstudy/m-p/167830#M4698 AdamIH123 2026-09-07T19:24:05Z