Difference between Workspace and Unity Catalog experiments when using MLflow autologging?

[kunduruanil]

Hi everyone,

I am trying to understand the exact differences between using Workspace experiments versus Unity Catalog experiments, specifically in the context of MLflow autologging (mlflow.autolog()).

Does autologging behave differently depending on whether the experiment is registered in the Workspace or in Unity Catalog (using a 3-level namespace)? Are there any limitations, best practices, or specific configurations I should be aware of when using autologging with Unity Catalog compared to the traditional Workspace setup?

Any insights or documentation links would be greatly appreciated. Thanks!

[gowri_databrick · Accepted solution]

@kunduruanil  Hi,

As far as I understand, mlflow.autolog() works in basically the same way with Workspace and Unity Catalog experiments. It automatically logs supported parameters, metrics, models, and other information from the ML training run.

The main difference is how the experiments and MLflow assets are managed and governed. Workspace experiments are tied to the workspace, while Unity Catalog gives you more centralized governance and access control.

For production use, Unity Catalog can be a better option when multiple teams need controlled access and governance.

I would also check the Databricks Runtime and MLflow versions, since some features and configurations may depend on the version you are using.

Hope this helps!

[kunduruanil]

@gowri_databrick  Mlflow version is 3.8.1 and I want to control how and where my experiments are stored!! Do you know how to do that?

[ivanvyd · Accepted solution]

@kunduruanil with MLflow 3.8.1, you control two storage decisions:

1. The tracking URI selects the server that stores experiment and run metadata.
2. The experiment's artifact location selects where MLflow stores model files and other artifacts.

mlflow.autolog() uses those settings. It does not create or select a separate Unity Catalog experiment.

For model-training runs, Databricks provides workspace experiments and notebook experiments. A three-level name such as catalog.schema.model_name identifies a registered model in Unity Catalog, not an experiment.

For the current Databricks workspace, configure a workspace experiment like this:

import mlflow

EXPERIMENT_NAME = "/Shared/ml-team/customer-churn"
ARTIFACT_LOCATION = (
"dbfs:/Volumes/main/mlops/mlflow_artifacts/customer_churn"
)

mlflow.set_tracking_uri("databricks")

if mlflow.get_experiment_by_name(EXPERIMENT_NAME) is None:
mlflow.create_experiment(
name=EXPERIMENT_NAME,
artifact_location=ARTIFACT_LOCATION,
)

mlflow.set_experiment(EXPERIMENT_NAME)
mlflow.autolog()

model.fit(X_train, y_train)

Databricks stores the parameters, metrics, tags, and run metadata in the workspace tracking server. The UC Volume contains the logged model files and other artifacts.

MLflow 3.8.1 supports this setup. Databricks requires MLflow 2.15 or later for experiment artifacts stored in UC Volumes.

An existing experiment keeps the artifact location chosen at creation. If the named experiment points somewhere else, create a new experiment with a new name and the required Volume path.

To send runs to another Databricks workspace, configure that workspace as a Databricks CLI profile and select it before setting the experiment:

mlflow.set_tracking_uri("databricks://<profile-name>")
mlflow.set_experiment("/Shared/ml-team/customer-churn")

Manage access to experiment metadata through workspace experiment permissions; manage access to artifact files through Unity Catalog grants on the Volume.

If you register the trained model, use a three-level name such as main.ml_models.customer_churn. MLflow 3 uses the Unity Catalog Model Rgeistry by default on supported Databricks workspaces. Model registration does not change where the source experiment lives.

Databricks also supports storing OpenTelemetry traces in UC Delta tables. That feature keeps the MLflow experiment as the UI entry point and requires MLflow 3.14 or later. It does not apply to model-training autologging with MLflow 3.8.1.