How could we share the Databricks ML runtime cluster among users when enable Unity Catalog

[ccsong]

Hi team,

Currently, we use the Databricks ML runtime to run our workflows and sometimes do the EDA. What we need is that we want to create a Databricks ML runtime for the team to share. When enabling Unity Catalog, how could we create a shared ML runtime to make the team members access the compute? Or what else do you suggest to us? Since it seems that the ML runtime is currently incompatible with the "Shared" mode.

Thanks a lot!

[Walter_C]

Hello many thanks for the question. As you have mentioned Shared Clusters currently do not have support for ML runtime, you can install the libraries you need manually on the non ML cluster to be used

[ccsong]

Thanks for your suggestion. And could you tell the concerns why ML rumtime does not support the Shared mode? And does the team have a plan to make the Shared Clusters support the ML runtime?

[Walter_C · Accepted solution]

Right now there is not plan to support ML runtime in shared clusters. Engineering is working on additional solutions but no ETA is currently available.

In regards why it is not supported, principal reason is due to isolation which is not available in shared clusters, especially for workloads that use distributed Machine Learning. This is a significant concern as it could potentially lead to data leaks or breaches.