DLT with unity catalog and ML
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-30-2023 06:50 AM
We are currently using DLT with unity catalog. DLT tables are created as materialized views in a schema inside a catalog.
When we try to access these materialized view using a ML runtime (ex. 13.0 ML) cluster, it says, that we must use Single User security mode. However, Single User security mode cannot access materialized views. It throws the error [MATERIALIZED_VIEW_OPRATION_NOT_ALLOWED.REQUIRES_SHARED_COMPUTE].
Is there any way to use DLT with unity catalog and ML all combined? We could create a notebook that copies the DLT materialized views into a Delta table but then there doesn't seem much of a point to using DLT.
Are we using DLT with Unity Catalog incorrectly? Should it only be used for bronze ingest/silver layer transformation and then we use Delta tables for gold layer tables?
- Labels:
-
DLT
-
ML
-
Unity Catalog
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-01-2024 07:54 AM
I recently hit the same issue.
Seems like this is a limitation of DLT with Unity Catalog.
Did you find a workaround @oteng? Otherwise I will try copying the materialized views to a table before doing the ML work.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-02-2024 07:45 AM
No workaround was found. We are just copying all the table to do the ML work. We haven't looked at this for a while though. So we are not aware of any new features.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-12-2024 12:20 AM
I managed to get some information from a friend at Databricks. Copying the tables in a separate workflow seems to be the best workaround for now.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-31-2024 05:46 AM
Is there any update on this? Basically you cannot access Materialized Views with ML Clsuters. To copy all tables for our Data Scientists seems like a really unnecessary step. Also they cannot profit from the advantage of the incremental table updates like others that can use shared cluster or SQL warehouses.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-03-2024 07:22 AM
No updates as far as I am aware.
You could make the workflow copying the data smart though and try to only do incremental updates, seems like a lot of effort though.