topic Re: DLT full refresh and resets in Data Engineering

DLT full refresh and resets

antr — Thu, 06 Feb 2025 09:00:49 GMT

When doing a full refresh in DLT, the ables seem to be in a reset/empty state until they're populated again. This can break downstream dependencies, if they try to use the data during pipeline execution.

How to handle such case properly?

Re: DLT full refresh and resets

antr — Thu, 06 Feb 2025 11:01:40 GMT

So that kind of approach would require additional logic outside of the DLT pipeline itself? Sounds doable, but the added complexity worries me, a lot.

Also, I don't understand why would I transfer data back to the main table because it's already populated in the end of refresh. Could you elaborate?

Re: DLT full refresh and resets

Advika — Wed, 12 Mar 2025 12:59:09 GMT

Hello @antr!

In DLT, a full refresh on a streaming table resets state processing and checkpoint data, potentially disrupting downstream processes that rely on it. To avoid this, use incremental updates (default) or append mode instead of full refreshes whenever possible.
Since DLT automatically repopulates tables after a refresh, schedule downstream jobs to run only after the pipeline completes to ensure data availability.

Re: DLT full refresh and resets

antr — Thu, 13 Mar 2025 06:57:20 GMT

> use incremental updates (default) or append mode instead of full refreshes whenever possible.

Yeah that is the default how we'll work. But because of DLT's nature, full-refreshes are quite common during changes in the pipeline, so they can't be avoided.

> Since DLT automatically repopulates tables after a refresh, schedule downstream jobs to run only after the pipeline completes to ensure data availability.

That sounds possible for simple cases, but in bigger environments this is just not viable as there are so many dependencies.

This is one thing more that makes DLT look like a half-baked solution.