topic mlflow RESOURCE_ALREADY_EXISTS in Data Engineering

mlflow RESOURCE_ALREADY_EXISTS

mangeldfz — Wed, 17 Nov 2021 09:13:35 GMT

I tried to log some run in my Databricks Workspace and I'm facing the following error: RESOURCE_ALREADY_EXISTS when I try to log any run.

I could replicate the error with the following code:

import mlflow
import mlflow.sklearn
from mlflow.tracking import MlflowClient
 
mlflow.set_experiment('/learning/Mlflow-Full-Example/test-mlflow')
 
with mlflow.start_run(run_name='silly_run-test') as run:
  mlflow.log_param('seed', 777)

The error is the following, I don't know what to do about the conflict with that AML experiment

In case the error image has not quality enough here is the full message:

RestException: RESOURCE_ALREADY_EXISTS: Failed to create AML experiment for experiment id=1823487114958629, name=/learning/Mlflow-Full-Example/test-mlflow, artifactLocation=dbfs:/databricks/mlflow-tracking/1823487114958629. There is an existing AML experiment with id=fa0eed6c-afd5-458b-9835-88903b535e04 and name='/adb/6432554542138879/1823487114958629/learning/Mlflow-Full-Example/test-mlflow' and artifactLocation='' that is not compatible.

Re: mlflow RESOURCE_ALREADY_EXISTS

mangeldfz — Wed, 17 Nov 2021 11:43:28 GMT

Hi Kaniz, thanks for your comment. I found another folk in the internet with the same problem starting a few days ago. So, I think it has nothing to do with my Workspace. Hope this will be solved soon, this is stopping all our machine learning developments.

Re: mlflow RESOURCE_ALREADY_EXISTS

Hubert-Dudek — Wed, 17 Nov 2021 12:10:08 GMT

it seems like name conflict can you just rename to something different than test-mlflow.

You can also try to clean directories if there is nothing important (but I am not sure is/ adb on dbfs storage):

dbutils.fs.rm("/databricks/mlflow-tracking/1823487114958629", recurse=True)
dbutils.fs.rm("/adb/6432554542138879/1823487114958629/learning/Mlflow-Full-Example/test-mlflow", recurse=True)

Re: mlflow RESOURCE_ALREADY_EXISTS

mangeldfz — Wed, 17 Nov 2021 12:49:19 GMT

I tried renaming the experiment name and the run_name and it does not work, the error keeps the same. When I search for the experiment it is conflicting I can find the AML id using client.list_experiments(), this is the Experiment I have conflict, but it seems the conflict has to do with the AML part:

Experiment: artifact_location='dbfs:/databricks/mlflow-tracking/2288118769165005', experiment_id='2288118769165005', lifecycle_stage='active', name='/learning/Mlflow-Full-Example/test-mlflow-renamed2', tags={'mlflow.AML_EXPERIMENT_ID': '594197a2-c16e-4e14-8040-e398833198ff',

'mlflow.experimentType': 'MLFLOW_EXPERIMENT',

'mlflow.ownerEmail': '***.***@***.***',

'mlflow.ownerId': 'YYYYYY'}

I can delete the whole experiment using the UI of Experiments if I try to delete any experiment using mlflow.delete_experiment() I get the same error of the beginning. Nevertheless, it does not work. Also, I can not find the /adb directory anywhere, it is not in the DBFS.

Re: mlflow RESOURCE_ALREADY_EXISTS

mangeldfz — Wed, 17 Nov 2021 13:01:43 GMT

It seems that for every experiment I create, mlflow creates also a AML experiment associated and all AML experiments are pointing to the same artifactLocation="" by default. It does not matter if you delete all experiments using the UI, the garbage collector detects that there is (or there was) a experiment (an AML experiment) with artifactLocation="", so there is a conflict for any new experiment you try to log things in.

Re: mlflow RESOURCE_ALREADY_EXISTS

Prabakar — Wed, 17 Nov 2021 16:10:24 GMT

Hi @Miguel Ángel Fernández it’s not recommended to “link” the Databricks and AML workspaces, as we are seeing more problems. You can refer to the instructions found below for using MLflow with AML. https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-mlflow

You can refer to https://github.com/MicrosoftDocs/azure-docs/issues/80298 to unlink.

Re: mlflow RESOURCE_ALREADY_EXISTS

mangeldfz — Thu, 18 Nov 2021 12:22:21 GMT

Hi Prabakar, thank you so much for your response. Finally, we decided to delete the Azure Machine Learning service because the ARM in the reference that you provide throws the following error:

I wonder if just redeploying the Azure Machine Learning service in the same resource group will be enough to set up both services properly or will be again a linking between them. I count with no mlflow communication between Databricks and the new Azure Machine Learning, of course.

Re: mlflow RESOURCE_ALREADY_EXISTS

Anonymous — Wed, 01 Dec 2021 16:01:26 GMT

Hi!

I am facing the same problems with linked WS and wonder if you managed to find a solution to your problem by unlinking the spaces.