topic Re: How could we share the Databricks ML runtime cluster among users when enable Unity Catalog in Administration & Architecture

How could we share the Databricks ML runtime cluster among users when enable Unity Catalog

ccsong — Wed, 26 Jun 2024 08:15:35 GMT

Hi team,

Currently, we use the Databricks ML runtime to run our workflows and sometimes do the EDA. What we need is that we want to create a Databricks ML runtime for the team to share. When enabling Unity Catalog, how could we create a shared ML runtime to make the team members access the compute? Or what else do you suggest to us? Since it seems that the ML runtime is currently incompatible with the "Shared" mode.

Thanks a lot!

Re: How could we share the Databricks ML runtime cluster among users when enable Unity Catalog

Walter_C — Sat, 29 Jun 2024 16:26:06 GMT

Hello many thanks for the question. As you have mentioned Shared Clusters currently do not have support for ML runtime, you can install the libraries you need manually on the non ML cluster to be used

Re: How could we share the Databricks ML runtime cluster among users when enable Unity Catalog

ccsong — Mon, 08 Jul 2024 05:43:25 GMT

Thanks for your suggestion. And could you tell the concerns why ML rumtime does not support the Shared mode? And does the team have a plan to make the Shared Clusters support the ML runtime?

Re: How could we share the Databricks ML runtime cluster among users when enable Unity Catalog

Walter_C — Tue, 09 Jul 2024 15:03:16 GMT

Right now there is not plan to support ML runtime in shared clusters. Engineering is working on additional solutions but no ETA is currently available.

In regards why it is not supported, principal reason is due to isolation which is not available in shared clusters, especially for workloads that use distributed Machine Learning. This is a significant concern as it could potentially lead to data leaks or breaches.