Unable to Check Experiment Existence with path starting with /Workspace/ Directory in Databricks Pla

Mikkel — Tue, 05 Nov 2024 09:15:04 GMT

https://github.com/mlflow/mlflow/issues/11077

In Databricks, when attempting to set an experiment with an experiment_name specified as an absolute path from /Workspace/Shared/mlflow_experiment/<experiment_name>, the mlflow.set_experiment() function succeeds in creating the experiment during the initial run.
However, upon subsequent attempts to set the experiment again using the same method, it results in an error.
I believe that this error occurs because the system fails to accurately check the existence of the experiment and mistakenly attempts to create it again with the same name, resulting in a RESOURCE_ALREADY_EXISTS error.

Re: Unable to Check Experiment Existence with path starting with /Workspace/ Directory in Databricks

Walter_C — Tue, 05 Nov 2024 14:20:03 GMT

Before setting the experiment, use mlflow.get_experiment_by_name() to check if the experiment already exists. If it does, you can set the experiment without attempting to create it again.

topic Unable to Check Experiment Existence with path starting with /Workspace/ Directory in Databricks Pla in Machine Learning

Unable to Check Experiment Existence with path starting with /Workspace/ Directory in Databricks Pla

Re: Unable to Check Experiment Existence with path starting with /Workspace/ Directory in Databricks