Combining DLT and workflow - MATERIALIZED_VIEW_OPERATION_NOT_ALLOWED

erigaud — Fri, 27 Oct 2023 07:04:48 GMT

Hello everyone !

I currently have a DLT pipeline that loads into several Delta LIVE tables (both streaming and materialized view).

The end table of my DLT pipeline is a materialized view called "silver.my_view".

In a later step I need to join/union/merge this table with an existing Delta Table (so not DLT).

To do so I followed my DLT pipeline with a regular notebook task in the same workflow, but I am met with the following error :

[MATERIALIZED_VIEW_OPERATION_NOT_ALLOWED.REQUIRES_SHARED_COMPUTE] The materialized view operation query is not allowed: Cannot query the Materialized View silver.my_view from an Assigned or No Isolation Shared cluster, please use a Shared cluster or a Databricks SQL warehouse instead.

I was using a regular job cluster for that. Is there a workaround ? Is using a Shared cluster the only way ? Aren't DLT intended to be used outside the DLT pipeline ?

Re: Combining DLT and workflow - MATERIALIZED_VIEW_OPERATION_NOT_ALLOWED

erigaud — Fri, 27 Oct 2023 11:22:12 GMT

Hello @Retired_mod ! Thanks for the answer ! The thing is I'm getting a similar error when trying to read streaming delta live tables.

Is there any way to read a table from a DLT pipeline with a regular non-shared cluster ? How should I define the table in my DLT pipeline so I can easily access it outside the pipeline ?

topic Combining DLT and workflow - MATERIALIZED_VIEW_OPERATION_NOT_ALLOWED in Data Engineering

Combining DLT and workflow - MATERIALIZED_VIEW_OPERATION_NOT_ALLOWED

Re: Combining DLT and workflow - MATERIALIZED_VIEW_OPERATION_NOT_ALLOWED