topic Re: Materialize View creation not supported from DLT Pipeline in Data Engineering

Materialize View creation not supported from DLT Pipeline

abhijitnag — Sun, 31 Dec 2023 18:41:50 GMT

Hi Team, I have a very basic scenario where I am using my custom catalog and want materialize view to get created from DLT table at the end of pipeline. The SQL used as below for the same.

where "loom_data_transform" is a Streaming table. But pipeline fails at the last stage where materialize view is tried to create from "loom_data_transform" with following error message.

Could someone please help me on this issue.

Re: Materialize View creation not supported from DLT Pipeline

warsamebashir — Mon, 01 Jan 2024 01:13:00 GMT

Hey @abhijitnag are you sure your loom_data_transform was created as a STREAMING table?

docs:

https://docs.databricks.com/en/sql/language-manual/sql-ref-syntax-ddl-create-streaming-table.html

Re: Materialize View creation not supported from DLT Pipeline

abhijitnag — Mon, 01 Jan 2024 02:46:17 GMT

Yes, @warsamebashir "loom_data_transform" was created as a STREAMING table as below. And without this Materialize view, the pipeline is executing absolutely fine.