โ06-26-2024 01:15 AM
Hi team,
Currently, we use the Databricks ML runtime to run our workflows and sometimes do the EDA. What we need is that we want to create a Databricks ML runtime for the team to share. When enabling Unity Catalog, how could we create a shared ML runtime to make the team members access the compute? Or what else do you suggest to us? Since it seems that the ML runtime is currently incompatible with the "Shared" mode.
Thanks a lot!
โ07-09-2024 08:03 AM
Right now there is not plan to support ML runtime in shared clusters. Engineering is working on additional solutions but no ETA is currently available.
In regards why it is not supported, principal reason is due to isolation which is not available in shared clusters, especially for workloads that use distributed Machine Learning. This is a significant concern as it could potentially lead to data leaks or breaches.
โ06-29-2024 09:26 AM
Hello many thanks for the question. As you have mentioned Shared Clusters currently do not have support for ML runtime, you can install the libraries you need manually on the non ML cluster to be used
โ07-07-2024 10:43 PM
Thanks for your suggestion. And could you tell the concerns why ML rumtime does not support the Shared mode? And does the team have a plan to make the Shared Clusters support the ML runtime?
โ07-09-2024 08:03 AM
Right now there is not plan to support ML runtime in shared clusters. Engineering is working on additional solutions but no ETA is currently available.
In regards why it is not supported, principal reason is due to isolation which is not available in shared clusters, especially for workloads that use distributed Machine Learning. This is a significant concern as it could potentially lead to data leaks or breaches.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group