https://community.databricks.com/t5/machine-learning/dlt-with-unity-catalog-and-ml/m-p/3880#M168 <P>We are currently using DLT with unity catalog. DLT tables are created as materialized views in a schema inside a catalog. </P><P></P><P>When we try to access these materialized view using a ML runtime (ex. 13.0 ML) cluster, it says, that we must use Single User security mode. However, Single User security mode cannot access materialized views. It throws the error [MATERIALIZED_VIEW_OPRATION_NOT_ALLOWED.REQUIRES_SHARED_COMPUTE].</P><P></P><P>Is there any way to use DLT with unity catalog and ML all combined? We could create a notebook that copies the DLT materialized views into a Delta table but then there doesn't seem much of a point to using DLT.</P><P></P><P>Are we using DLT with Unity Catalog incorrectly? Should it only be used for bronze ingest/silver layer transformation and then we use Delta tables for gold layer tables?</P> Tue, 30 May 2023 13:50:15 GMT oteng 2023-05-30T13:50:15Z https://community.databricks.com/t5/machine-learning/dlt-with-unity-catalog-and-ml/m-p/3880#M168 <P>We are currently using DLT with unity catalog. DLT tables are created as materialized views in a schema inside a catalog. </P><P></P><P>When we try to access these materialized view using a ML runtime (ex. 13.0 ML) cluster, it says, that we must use Single User security mode. However, Single User security mode cannot access materialized views. It throws the error [MATERIALIZED_VIEW_OPRATION_NOT_ALLOWED.REQUIRES_SHARED_COMPUTE].</P><P></P><P>Is there any way to use DLT with unity catalog and ML all combined? We could create a notebook that copies the DLT materialized views into a Delta table but then there doesn't seem much of a point to using DLT.</P><P></P><P>Are we using DLT with Unity Catalog incorrectly? Should it only be used for bronze ingest/silver layer transformation and then we use Delta tables for gold layer tables?</P> Tue, 30 May 2023 13:50:15 GMT https://community.databricks.com/t5/machine-learning/dlt-with-unity-catalog-and-ml/m-p/3880#M168 oteng 2023-05-30T13:50:15Z https://community.databricks.com/t5/machine-learning/dlt-with-unity-catalog-and-ml/m-p/58991#M2937 <P>I recently hit the same issue.<BR />Seems like this is a limitation of DLT with Unity Catalog.<BR /><BR />Did you find a workaround&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/29629">@oteng</a>? Otherwise I will try copying the materialized views to a table before doing the ML work.</P> Thu, 01 Feb 2024 15:54:18 GMT https://community.databricks.com/t5/machine-learning/dlt-with-unity-catalog-and-ml/m-p/58991#M2937 pg5 2024-02-01T15:54:18Z https://community.databricks.com/t5/machine-learning/dlt-with-unity-catalog-and-ml/m-p/59069#M2941 <P>No workaround was found. We are just copying all the table to do the ML work. We haven't looked at this for a while though. So we are not aware of any new features.</P> Fri, 02 Feb 2024 15:45:29 GMT https://community.databricks.com/t5/machine-learning/dlt-with-unity-catalog-and-ml/m-p/59069#M2941 oteng 2024-02-02T15:45:29Z https://community.databricks.com/t5/machine-learning/dlt-with-unity-catalog-and-ml/m-p/59899#M2990 <P>I managed to get some information from a friend at Databricks. Copying the tables in a separate workflow seems to be the best workaround for now.</P> Mon, 12 Feb 2024 08:20:11 GMT https://community.databricks.com/t5/machine-learning/dlt-with-unity-catalog-and-ml/m-p/59899#M2990 pg5 2024-02-12T08:20:11Z https://community.databricks.com/t5/machine-learning/dlt-with-unity-catalog-and-ml/m-p/71190#M3312 <P>Is there any update on this? Basically you cannot access Materialized Views with ML Clsuters. To copy all tables for our Data Scientists seems like a really unnecessary step. Also they cannot profit from the advantage of the incremental table updates like others that can use shared cluster or SQL warehouses.</P> Fri, 31 May 2024 12:46:22 GMT https://community.databricks.com/t5/machine-learning/dlt-with-unity-catalog-and-ml/m-p/71190#M3312 MarkusFra 2024-05-31T12:46:22Z https://community.databricks.com/t5/machine-learning/dlt-with-unity-catalog-and-ml/m-p/71487#M3327 <P>No updates as far as I am aware.<BR />You could make the workflow copying the data smart though and try to only do incremental updates, seems like a lot of effort though.</P> Mon, 03 Jun 2024 14:22:17 GMT https://community.databricks.com/t5/machine-learning/dlt-with-unity-catalog-and-ml/m-p/71487#M3327 pg5 2024-06-03T14:22:17Z