Clarification on Databricks Access Modes: Standard vs. Dedicated with Unity Catalog

[Yuki]

Hello,

I’d like to ask for clarification regarding the access modes in Databricks, specifically the intent and future direction of the “Standard” and “Dedicated” modes.

According to the documentation below:
https://docs.databricks.com/aws/ja/compute/access-mode-limitations

It recommends using the Standard access mode (formerly Shared access mode) for most workloads. However, I find it a bit confusing that under the section “Limitations of Standard access mode with Unity Catalog,” it states that Databricks Runtime ML is not supported.

As I’ve started getting used to Unity Catalog, I find the Standard mode easier to manage permissions in a way that aligns with groups and accounts. However, if I want to use ML Runtime, I’m required to switch to Dedicated mode. This raises some concerns, as it means I need to manually configure permissions for folders containing imported libraries at the user or group level, and also manage Secrets separately.

I apologize if I’m missing some background context, as I’m not an admin and may not fully understand the historical reasons behind these design choices.

Any insights or guidance would be greatly appreciated.

Thank you!

[Vidhi_Khaitan · Accepted solution]

Hi Yuki,

The primary reason for ML runtime not being supported on Shared mode cluster is security and resource isolation: 1) ML workloads frequently require privileged operations (e.g., running arbitrary code, installing dependencies) not compatible with the multi-user process isolation model of Standard mode.
2) Many ML libraries (especially GPU-enabled or native code) need access to the underlying filesystem or privileged resources, which could break the isolation guarantees required for data security/governance in Standard mode.
3) Supporting ML Runtime in Standard would open up nontrivial risk of privilege escalation or governance circumvention, and enforcement is difficult.

Whereas,

Dedicated mode provides each compute resource to a single principal (user or group). This mode:
1) Allows installation of arbitrary libraries, use of MLflow, custom environments, GPU acceleration, and access to features like DBFS/FUSE that multi-user safety would otherwise restrict.
2) Makes ML workloads possible while still integrating with Unity Catalog for data governance—albeit at the cost of simplified sharing and more manual folder/secret management.

I hope this clarifies your question!

[Yuki]

Hi @Vidhi_Khaitan ,

Many thanks for your explanation. That's perfect for me.

I never thought about it and realized the isolations.
And I understand that the Dedicated cluster play important role due to these many important reasons.

I felt Databricks made a great effort to manage compute resources despite the complex data governance requirements.

Thank you for your great response!

[Vidhi_Khaitan]

Hi @Yuki Thank you, I am glad I could help!