Delta Live Table pipeline failure - Table missing

Manjula_Ganesap
Contributor

‎08-18-2023 08:35 AM - edited ‎08-18-2023 08:42 AM

Hi All,

I set up a DLT pipeline to create 58 bronze tables and a subsequent DLT live table that joins the 58 bronze tables created in the first step. The pipeline runs successfully most times.

My issue is that the pipeline fails once every 3/4 runs saying that a table is missing - but when i rerun without any code change - it completes successfully. I'm not sure what is happening.

I face the issue - both while running from UI and ADP (full refresh enabled)

Manjula_Ganesap_0-1692373291621.pngManjula_Ganesap_1-1692373340027.png

from the screenshot: both of the tables are created in the first step.

Any guidance would be really helpful and appreciated. Thank you. 

0 Kudos

jose_gonzalez
Databricks Employee
Databricks Employee

‎09-08-2023 03:50 PM

There is a blank space in your table's name "test_dlt_<blank_space>" could you check your tables? is the name correct?

0 Kudos

Manjula_Ganesap
Contributor

‎09-21-2023 07:10 AM

@jose_gonzalez @Retired_mod  - Missed to update the group on the fix. Reached out to Databricks to understand and it was identified that the threads call that i was making was causing the issue. After i removed it - i don't see it happening. 

View solution in original post