cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How could we share the Databricks ML runtime cluster among users when enable Unity Catalog

ccsong
New Contributor II

Hi team,

Currently, we use the Databricks ML runtime to run our workflows and sometimes do the EDA. What we need is that we want to create a Databricks ML runtime for the team to share. When enabling Unity Catalog, how could we create a shared ML runtime to make the team members access the compute? Or what else do you suggest to us? Since it seems that the ML runtime is currently incompatible with the "Shared" mode.

Thanks a lot!

1 ACCEPTED SOLUTION

Accepted Solutions

Walter_C
Databricks Employee
Databricks Employee

Right now there is not plan to support ML runtime in shared clusters. Engineering is working on additional solutions but no ETA is currently available.

In regards why it is not supported, principal reason is due to isolation which is not available in shared clusters, especially for workloads that use distributed Machine Learning. This is a significant concern as it could potentially lead to data leaks or breaches.

 

View solution in original post

3 REPLIES 3

Walter_C
Databricks Employee
Databricks Employee

Hello many thanks for the question. As you have mentioned Shared Clusters currently do not have support for ML runtime, you can install the libraries you need manually on the non ML cluster to be used

 

ccsong
New Contributor II

Thanks for your suggestion. And could you tell the concerns why ML rumtime does not support the Shared mode? And does the team have a plan to make the Shared Clusters support the ML runtime?

Walter_C
Databricks Employee
Databricks Employee

Right now there is not plan to support ML runtime in shared clusters. Engineering is working on additional solutions but no ETA is currently available.

In regards why it is not supported, principal reason is due to isolation which is not available in shared clusters, especially for workloads that use distributed Machine Learning. This is a significant concern as it could potentially lead to data leaks or breaches.

 

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now