- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-26-2024 01:15 AM
Hi team,
Currently, we use the Databricks ML runtime to run our workflows and sometimes do the EDA. What we need is that we want to create a Databricks ML runtime for the team to share. When enabling Unity Catalog, how could we create a shared ML runtime to make the team members access the compute? Or what else do you suggest to us? Since it seems that the ML runtime is currently incompatible with the "Shared" mode.
Thanks a lot!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-29-2024 09:26 AM
Hello many thanks for the question. As you have mentioned Shared Clusters currently do not have support for ML runtime, you can install the libraries you need manually on the non ML cluster to be used
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-07-2024 10:43 PM
Thanks for your suggestion. And could you tell the concerns why ML rumtime does not support the Shared mode? And does the team have a plan to make the Shared Clusters support the ML runtime?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-09-2024 08:03 AM
Right now there is not plan to support ML runtime in shared clusters. Engineering is working on additional solutions but no ETA is currently available.
In regards why it is not supported, principal reason is due to isolation which is not available in shared clusters, especially for workloads that use distributed Machine Learning. This is a significant concern as it could potentially lead to data leaks or breaches.
