mark_ott
Databricks Employee
Databricks Employee

‎10-24-2025 07:21 AM

The error

text
OSError: [Errno 5] Input/output error: '/dbfs/Volumes'

occurs because Databricks Apps (including Streamlit apps running on Databricks) currently do not have direct write access to /dbfs/Volumes for artifact logging via MLflow within the app execution environment.​

Here are the main causes and solutions you can apply:

Why It Happens

  1. Permissions issue with Volumes

    • Unity Catalog Volumes require specific privileges: USE CATALOG, USE SCHEMA, and USE VOLUME.

    • Your Databricks App likely runs under a service principal or app environment that does not have write access to the target volume.​

  2. DBFS and Volume availability in Databricks Apps

    • In the Databricks Apps runtime, paths like /dbfs/Volumes/... may not be mounted or accessible as standard local paths for direct file operations. This leads to input/output errors even when the volume exists.​

  3. Custom artifact location

    • When using a custom artifact_location (e.g., dbfs:/Volumes/...), you must ensure MLflow supports that location and your client version is ≥ 2.15.0. Otherwise, MLflow may misinterpret the artifact root and fail to write files.​

Fix: Recommended Approaches

Option 1: Use MLflow-managed storage

Remove the custom artifact location and let MLflow assign the default tracked location:

python
experiment_name = "/Shared/test_run" mlflow.set_experiment(experiment_name)

Databricks-managed artifact storage automatically applies proper permissions (stored under dbfs:/databricks/mlflow-tracking/…).​

Option 2: Use a Unity Catalog Volume with proper permissions

If you must use a specific Volume:

  1. Ensure your catalog/schema/volume exist and grant privileges:

    sql
    GRANT USE CATALOG ON CATALOG testing TO `your_user_or_service_principal`; GRANT USE SCHEMA ON SCHEMA testing.model-data TO `your_user_or_service_principal`; GRANT USE VOLUME ON VOLUME testing.model-data.training-data TO `your_user_or_service_principal`;

  2. Use a recent MLflow version (≥ 2.15.0):

    python
    %pip install --upgrade mlflow

  3. Set artifact_location as:

    python
    artifact_location = f"dbfs:/Volumes/{catalog}/{schema}/{volume}/mlflow_artifacts"

Option 3: Log to temporary local storage, then move manually

Log artifacts to a local temp directory like /tmp and copy them afterward:

python
mlflow.log_artifact(temp_csv_path, artifact_path="data")

Follow with:

python
dbutils.fs.cp("file:/tmp/mmm_data.csv", f"dbfs:/Volumes/{catalog}/{schema}/{volume}/mmm_data.csv")

Summary of Key Recommendations

Cause Fix
Lack of permissions on Unity Catalog volume Grant USE VOLUME permissions
Inaccessible /dbfs within Databricks Apps Use dbutils.fs.cp or default MLflow artifact store
Outdated MLflow version Upgrade to ≥ 2.15.0
Custom artifact location not supported Prefer default MLflow-managed locations
 
 

Following these steps will allow your Streamlit app in Databricks to log artifacts without hitting the OSError: [Errno 5] Input/output error.

View solution in original post