08-18-2023 08:35 AM - edited 08-18-2023 08:42 AM
Hi All,
I set up a DLT pipeline to create 58 bronze tables and a subsequent DLT live table that joins the 58 bronze tables created in the first step. The pipeline runs successfully most times.
My issue is that the pipeline fails once every 3/4 runs saying that a table is missing - but when i rerun without any code change - it completes successfully. I'm not sure what is happening.
I face the issue - both while running from UI and ADP (full refresh enabled)
from the screenshot: both of the tables are created in the first step.
Any guidance would be really helpful and appreciated. Thank you.
09-21-2023 07:10 AM
@jose_gonzalez @Kaniz_Fatma - Missed to update the group on the fix. Reached out to Databricks to understand and it was identified that the threads call that i was making was causing the issue. After i removed it - i don't see it happening.
09-06-2023 04:05 PM - edited 09-06-2023 04:06 PM
Hi @Manjula_Ganesap ,
Possible reasons for DLT pipeline failure
Table Dependency: Pipeline may depend on the previous pipeline or specific table completion. Check dependencies and ensure required tables are created before running the pipeline.
• Data Availability: Data may not be consistently available from external sources. Verify data source availability and accessibility during pipeline runs.
• Pipeline Configuration: Configuration issues may cause intermittent failures. Review pipeline configuration and ensure correct settings.
Steps to troubleshoot the issue:
• Check logs: Review logs from failed pipeline runs for error messages or warnings.
• Rerun without code change: If rerunning the pipeline without code change resolves the issue, focus on factors like data availability or pipeline configuration.
• Monitor system resources: Check resource utilization (CPU, memory, disk) during pipeline runs. Optimize resource allocation if the pipeline exceeds allocated resources.
09-08-2023 03:50 PM
There is a blank space in your table's name "test_dlt_<blank_space>" could you check your tables? is the name correct?
09-21-2023 07:10 AM
@jose_gonzalez @Kaniz_Fatma - Missed to update the group on the fix. Reached out to Databricks to understand and it was identified that the threads call that i was making was causing the issue. After i removed it - i don't see it happening.
09-21-2023 08:28 AM
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group