topic Re: Issue with DLT Pipelines in Data Engineering

Issue with DLT Pipelines

ash_pal — Thu, 15 Aug 2024 00:26:14 GMT

Hi Team,

We are trying to create DLT pipeline. The scenario is this:-

We already have catalog in unity catalog and under that we have schema called test in which we have 17 tables. Now we are trying to create DLT pipeline and copy the data from those 17 table from test schema to another prd schema.

I already create notebook in which we have this statement

CREATE OR REFRESH STREAMING TABLE prd_acct

as select * from STREAM(catalogname_test.acct);

And same SQL command for all 17 tables. Now when i run the DLT pipeline all the 17 tables has been created in prd schema. However, when graph appear it show couple of tables has been failed with this error.

And when i check the prd schema all the tables appears correctly .

I run the DLT job against cluster only.

Has anyone have any idea why this error happened to me OR what will be correct way to run the DLT pipeline.

Any help will be really appreciated.

Re: Issue with DLT Pipelines

jessysantos — Mon, 19 Aug 2024 15:00:20 GMT

Hey @ash_pal!

When you click on the "flow_progress" error in the screenshot you shared, does a pop-up window appear showing the Summary of the pipeline event along with the Error Details? If so, please provide us with the Error details so we can investigate further.

Best regards,

Jéssica Santos

Re: Issue with DLT Pipelines

ash_pal — Mon, 19 Aug 2024 22:17:23 GMT

Hi Jessy,

Thanks for reply. Please find the error message below