Combining DLT and workflow - MATERIALIZED_VIEW_OPERATION_NOT_ALLOWED
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-27-2023 12:04 AM
Hello everyone !
I currently have a DLT pipeline that loads into several Delta LIVE tables (both streaming and materialized view).
The end table of my DLT pipeline is a materialized view called "silver.my_view".
In a later step I need to join/union/merge this table with an existing Delta Table (so not DLT).
To do so I followed my DLT pipeline with a regular notebook task in the same workflow, but I am met with the following error :
[MATERIALIZED_VIEW_OPERATION_NOT_ALLOWED.REQUIRES_SHARED_COMPUTE] The materialized view operation query is not allowed: Cannot query the Materialized View silver.my_view from an Assigned or No Isolation Shared cluster, please use a Shared cluster or a Databricks SQL warehouse instead.
I was using a regular job cluster for that. Is there a workaround ? Is using a Shared cluster the only way ? Aren't DLT intended to be used outside the DLT pipeline ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-27-2023 04:22 AM
Hello @Retired_mod ! Thanks for the answer ! The thing is I'm getting a similar error when trying to read streaming delta live tables.
Is there any way to read a table from a DLT pipeline with a regular non-shared cluster ? How should I define the table in my DLT pipeline so I can easily access it outside the pipeline ?
