topic Delta Live Table pipeline failure - Table missing in Data Engineering

Delta Live Table pipeline failure - Table missing

Manjula_Ganesap — Fri, 18 Aug 2023 15:42:50 GMT

Hi All,

I set up a DLT pipeline to create 58 bronze tables and a subsequent DLT live table that joins the 58 bronze tables created in the first step. The pipeline runs successfully most times.

My issue is that the pipeline fails once every 3/4 runs saying that a table is missing - but when i rerun without any code change - it completes successfully. I'm not sure what is happening.

I face the issue - both while running from UI and ADP (full refresh enabled)

from the screenshot: both of the tables are created in the first step.

Any guidance would be really helpful and appreciated. Thank you.

Re: Delta Live Table pipeline failure - Table missing

jose_gonzalez — Fri, 08 Sep 2023 22:50:58 GMT

There is a blank space in your table's name "test_dlt_<blank_space>" could you check your tables? is the name correct?

Re: Delta Live Table pipeline failure - Table missing

Manjula_Ganesap — Thu, 21 Sep 2023 14:10:05 GMT

@jose_gonzalez @Retired_mod - Missed to update the group on the fix. Reached out to Databricks to understand and it was identified that the threads call that i was making was causing the issue. After i removed it - i don't see it happening.