Hi,

I am trying to follow this simple document to be able to run MLFlow within Databricks:

https://docs.microsoft.com/en-us/azure/databricks/applications/mlflow/projects

I try to run it from:

1. A Databricks notebook within Azure Databricks
2. By use of the mlflow-cli (remote)
3. By use of databricks-connect

I have tested that the 3 methods are properly set-up. I get the same error with all methods:

mlflow.exceptions.RestException: BAD_REQUEST: Unable to connect to the linked AzureML workspace. Check that the workspace exists.

The Databricks workspace is linked to an AzureML workspace.

By following this other document:

https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-mlflow-azure-databricks

I am actually able to create and run MLFlow experiments from:

1. A Databricks notebook
2. databricks-connect

The issue that I have is that:

1. The experiments are only created in AzureML
2. I can only run from within a script/notebook

If I follow this document:

https://docs.microsoft.com/en-us/azure/databricks/applications/mlflow/access-hosted-tracking-server

I am able to use the remote mlflow-cli and I can for example, create an experiment in databricks only (the experiment doesn't live in AzureML), by use of:

mlflow experiments create -n /Users/<your-username>/my-experiment

But again, when trying to do something like this:

mlflow run https://github.com/mlflow/mlflow#examples/sklearn_elasticnet_wine -b databricks --backend-config cluster-spec.json --experiment-id <experiment-id>

I get the error I previously mentioned:

mlflow.exceptions.RestException: BAD_REQUEST: Unable to connect to the linked AzureML workspace. Check that the workspace exists.

I have set

export MLFLOW_TRACKING_URI=databricks

And everything else as noted in the documentation.

Is there a configuration I am missing?

Thanks a lot in advance for any help!