cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Issue with DLT Pipelines

ash_pal
New Contributor II

Hi Team,

We are trying to create DLT pipeline. The scenario is this:- 

We already have catalog in unity catalog and under that we have schema called test in which we have 17 tables. Now we are trying to create DLT pipeline and copy the data from those 17 table from test schema to another prd schema. 

I already create notebook in which we have this statement 

CREATE OR REFRESH STREAMING TABLE prd_acct
as select * from STREAM(catalogname_test.acct);
 
And same SQL command for all 17 tables. Now when i run the DLT pipeline all the 17 tables has been created in prd schema. However, when graph appear it show couple of tables has been failed with this error.
ash_pal_0-1723681443979.png

And when i check the prd schema all the tables appears correctly . 

I run the DLT job against cluster only.

Has anyone have any idea why this error happened to me OR what will be correct way to run the DLT pipeline.

Any help will be really appreciated. 

2 REPLIES 2

jessysantos
Databricks Employee
Databricks Employee

Hey @ash_pal!

When you click on the "flow_progress" error in the screenshot you shared, does a pop-up window appear showing the Summary of the pipeline event along with the Error Details? If so, please provide us with the Error details so we can investigate further.

Best regards, 

Jéssica Santos

ash_pal
New Contributor II

Hi Jessy,

Thanks for reply. Please find the error message below

ash_pal_0-1724105811718.png

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group