topic Re: Delta Live Table missing data in Data Engineering

Delta Live Table missing data

tommyhmt — Thu, 07 Nov 2024 09:45:40 GMT

Got a very simple DLT which runs fine, but the final table "a" is missing data.

I've found that after goes through a full refresh, if I rerun just the final table, then I get more records (from 1.2m to 1.4m) and the missing data then comes back.

When I run the last bit of the code in a notebook manually then I do get all the records, so I don't know where the 1.2m comes from. I feel like it's somehow reading an older version of the r and c tables, but that shouldn't be happening if it's a full refresh right?

I've also noticed that I get the following warning:

Your query 'r' reads from '<catalog name>.<schema name>.p' but must read from 'LIVE.p' instead. Always use the LIVE keyword when referencing tables from the same pipeline so that DLT can track the dependencies in the pipeline.

But in the code in my notebook I already use LIVE. and don't reference any catalog or schema.

Re: Delta Live Table missing data

NSonam — Thu, 09 Jan 2025 02:39:39 GMT

To me it seems like timing or dependency issue. The missing data could be due to intermediate tables are not being properly refreshed or triggered during the full refresh. Please check if intermediate tables are being loaded properly before it start executing last bit of code.

Re: Delta Live Table missing data

tommyhmt — Thu, 06 Mar 2025 12:35:43 GMT

this issue was found in DEV and so there is no new data being added/removed, furthermore I am able to create the screenshot above because everytime I do a full refresh I get 1.2m, and everytime I refresh just the final table I get 1.4m