mlflow RESOURCE_ALREADY_EXISTS

[mangeldfz]

I tried to log some run in my Databricks Workspace and I'm facing the following error: RESOURCE_ALREADY_EXISTS when I try to log any run.

I could replicate the error with the following code:

import mlflow
import mlflow.sklearn
from mlflow.tracking import MlflowClient
 
mlflow.set_experiment('/learning/Mlflow-Full-Example/test-mlflow')
 
with mlflow.start_run(run_name='silly_run-test') as run:
  mlflow.log_param('seed', 777)

The error is the following, I don't know what to do about the conflict with that AML experiment

In case the error image has not quality enough here is the full message:

RestException: RESOURCE_ALREADY_EXISTS: Failed to create AML experiment for experiment id=1823487114958629, name=/learning/Mlflow-Full-Example/test-mlflow, artifactLocation=dbfs:/databricks/mlflow-tracking/1823487114958629. There is an existing AML experiment with id=fa0eed6c-afd5-458b-9835-88903b535e04 and name='/adb/6432554542138879/1823487114958629/learning/Mlflow-Full-Example/test-mlflow' and artifactLocation='' that is not compatible.

[mangeldfz]

Hi Kaniz, thanks for your comment. I found another folk in the internet with the same problem starting a few days ago. So, I think it has nothing to do with my Workspace. Hope this will be solved soon, this is stopping all our machine learning developments.

[Hubert-Dudek]

it seems like name conflict can you just rename to something different than test-mlflow.

You can also try to clean directories if there is nothing important (but I am not sure is/ adb on dbfs storage):

dbutils.fs.rm("/databricks/mlflow-tracking/1823487114958629", recurse=True)
dbutils.fs.rm("/adb/6432554542138879/1823487114958629/learning/Mlflow-Full-Example/test-mlflow", recurse=True)

[mangeldfz]

I tried renaming the experiment name and the run_name and it does not work, the error keeps the same. When I search for the experiment it is conflicting I can find the AML id using client.list_experiments(), this is the Experiment I have conflict, but it seems the conflict has to do with the AML part:

Experiment: artifact_location='dbfs:/databricks/mlflow-tracking/2288118769165005', experiment_id='2288118769165005', lifecycle_stage='active', name='/learning/Mlflow-Full-Example/test-mlflow-renamed2', tags={'mlflow.AML_EXPERIMENT_ID': '594197a2-c16e-4e14-8040-e398833198ff',

'mlflow.experimentType': 'MLFLOW_EXPERIMENT',

'mlflow.ownerEmail': '***.***@***.***',

'mlflow.ownerId': 'YYYYYY'}

I can delete the whole experiment using the UI of Experiments if I try to delete any experiment using mlflow.delete_experiment() I get the same error of the beginning. Nevertheless, it does not work. Also, I can not find the /adb directory anywhere, it is not in the DBFS.

[mangeldfz]

It seems that for every experiment I create, mlflow creates also a AML experiment associated and all AML experiments are pointing to the same artifactLocation="" by default. It does not matter if you delete all experiments using the UI, the garbage collector detects that there is (or there was) a experiment (an AML experiment) with artifactLocation="", so there is a conflict for any new experiment you try to log things in.

[Prabakar · Accepted solution]

Hi @Miguel Ángel Fernández​  it’s not recommended to “link” the Databricks and AML workspaces, as we are seeing more problems. You can refer to the instructions found below for using MLflow with AML.   https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-mlflow

You can refer to https://github.com/MicrosoftDocs/azure-docs/issues/80298 to unlink.

[mangeldfz]

Hi Prabakar, thank you so much for your response. Finally, we decided to delete the Azure Machine Learning service because the ARM in the reference that you provide throws the following error:

 

I wonder if just redeploying the Azure Machine Learning service in the same resource group will be enough to set up both services properly or will be again a linking between them. I count with no mlflow communication between Databricks and the new Azure Machine Learning, of course.

[Anonymous]

Hi!

I am facing the same problems with linked WS and wonder if you managed to find a solution to your problem by unlinking the spaces.