Unable to Check Experiment Existence with path starting with /Workspace/ Directory in Databricks Pla
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-05-2024 01:15 AM
https://github.com/mlflow/mlflow/issues/11077
In Databricks, when attempting to set an experiment with an experiment_name specified as an absolute path from /Workspace/Shared/mlflow_experiment/<experiment_name>, the mlflow.set_experiment() function succeeds in creating the experiment during the initial run.
However, upon subsequent attempts to set the experiment again using the same method, it results in an error.
I believe that this error occurs because the system fails to accurately check the existence of the experiment and mistakenly attempts to create it again with the same name, resulting in a RESOURCE_ALREADY_EXISTS error.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-05-2024 06:20 AM
Before setting the experiment, use mlflow.get_experiment_by_name() to check if the experiment already exists. If it does, you can set the experiment without attempting to create it again.
