DLT full refresh and resets

[antr]

When doing a full refresh in DLT, the ables seem to be in a reset/empty state until they're populated again. This can break downstream dependencies, if they try to use the data during pipeline execution.

How to handle such case properly?

[antr]

So that kind of approach would require additional logic outside of the DLT pipeline itself? Sounds doable, but the added complexity worries me, a lot. 

Also, I don't understand why would I transfer data back to the main table because it's already populated in the end of refresh. Could you elaborate?

[Advika]

Hello @antr!

In DLT, a full refresh on a streaming table resets state processing and checkpoint data, potentially disrupting downstream processes that rely on it. To avoid this, use incremental updates (default) or append mode instead of full refreshes whenever possible.
Since DLT automatically repopulates tables after a refresh, schedule downstream jobs to run only after the pipeline completes to ensure data availability.

[antr]

> use incremental updates (default) or append mode instead of full refreshes whenever possible.

Yeah that is the default how we'll work. But because of DLT's nature, full-refreshes are quite common during changes in the pipeline, so they can't be avoided.
 

> Since DLT automatically repopulates tables after a refresh, schedule downstream jobs to run only after the pipeline completes to ensure data availability.

That sounds possible for simple cases, but in bigger environments this is just not viable as there are so many dependencies.

This is one thing more that makes DLT look like a half-baked solution.