cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT with unity catalog and ML

oteng
New Contributor III

We are currently using DLT with unity catalog. DLT tables are created as materialized views in a schema inside a catalog.

When we try to access these materialized view using a ML runtime (ex. 13.0 ML) cluster, it says, that we must use Single User security mode. However, Single User security mode cannot access materialized views. It throws the error [MATERIALIZED_VIEW_OPRATION_NOT_ALLOWED.REQUIRES_SHARED_COMPUTE].

Is there any way to use DLT with unity catalog and ML all combined? We could create a notebook that copies the DLT materialized views into a Delta table but then there doesn't seem much of a point to using DLT.

Are we using DLT with Unity Catalog incorrectly? Should it only be used for bronze ingest/silver layer transformation and then we use Delta tables for gold layer tables?

3 REPLIES 3

pg5
New Contributor II

I recently hit the same issue.
Seems like this is a limitation of DLT with Unity Catalog.

Did you find a workaround @oteng? Otherwise I will try copying the materialized views to a table before doing the ML work.

oteng
New Contributor III

No workaround was found. We are just copying all the table to do the ML work. We haven't looked at this for a while though. So we are not aware of any new features.

pg5
New Contributor II

I managed to get some information from a friend at Databricks. Copying the tables in a separate workflow seems to be the best workaround for now.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.