Delta Live Table pipeline failure - Table missing

[Manjula_Ganesap]

Hi All,

I set up a DLT pipeline to create 58 bronze tables and a subsequent DLT live table that joins the 58 bronze tables created in the first step. The pipeline runs successfully most times.

My issue is that the pipeline fails once every 3/4 runs saying that a table is missing - but when i rerun without any code change - it completes successfully. I'm not sure what is happening.

I face the issue - both while running from UI and ADP (full refresh enabled)

from the screenshot: both of the tables are created in the first step.

Any guidance would be really helpful and appreciated. Thank you. 

[Kaniz_Fatma]

Hi @Manjula_Ganesap , 

Possible reasons for DLT pipeline failure 

Table Dependency: Pipeline may depend on the previous pipeline or specific table completion. Check dependencies and ensure required tables are created before running the pipeline.

• Data Availability: Data may not be consistently available from external sources. Verify data source availability and accessibility during pipeline runs.

• Pipeline Configuration: Configuration issues may cause intermittent failures. Review pipeline configuration and ensure correct settings.

Steps to troubleshoot the issue:

• Check logs: Review logs from failed pipeline runs for error messages or warnings.

• Rerun without code change: If rerunning the pipeline without code change resolves the issue, focus on factors like data availability or pipeline configuration.

• Monitor system resources: Check resource utilization (CPU, memory, disk) during pipeline runs. Optimize resource allocation if the pipeline exceeds allocated resources.

[jose_gonzalez]

There is a blank space in your table's name "test_dlt_<blank_space>" could you check your tables? is the name correct?

[Manjula_Ganesap · Accepted solution]

@jose_gonzalez @Kaniz_Fatma  - Missed to update the group on the fix. Reached out to Databricks to understand and it was identified that the threads call that i was making was causing the issue. After i removed it - i don't see it happening. 

[Kaniz_Fatma]

Awesome, Thanks for the update 😃.