When doing a full refresh in DLT, the ables seem to be in a reset/empty state until they're populated again. This can break downstream dependencies, if they try to use the data during pipeline execution.How to handle such case properly?
In Delta Live Tables, the INITIALIZING phase takes sometimes a minute, sometimes 5 minutes. I'd like to learn what is it doing in the background, and can it be optimized in any way.
> use incremental updates (default) or append mode instead of full refreshes whenever possible.Yeah that is the default how we'll work. But because of DLT's nature, full-refreshes are quite common during changes in the pipeline, so they can't be avoi...
So that kind of approach would require additional logic outside of the DLT pipeline itself? Sounds doable, but the added complexity worries me, a lot. Also, I don't understand why would I transfer data back to the main table because it's already popu...
Thanks for the answer. I'm a bit confused as SETTING_UP_TABLES is a separate phase after INITIALIZING, and you say "Setting up all tables" is part of initialization. Could you elaborate how they differ?
