DLT full refresh

[PassionateDBD]

Running a task with full refresh in delta live tables removes existing data and reloads it from scratch. We are ingesting data from an event hub topic and from files. The event hub topic stores messages for seven days after arrival. If we would run a full refresh at some point in the future then wouldn't we lose all data except from the previous seven days as older data cannot be loaded anymore from the event hub topic?

If my interpretation is correct then is there a way to "mark" tables so that they are the "original source" of data and they should not be recalculated even if we run a full refresh? I would assume that this is a common case that bronze+silver+gold layers would need to be reloaded but landing would keep the full history of data in original form. 

[Kaniz]

Hi @PassionateDBD, About preserving the full history of data, consider marking certain tables as the “original source.”

• Unfortunately, there isn’t a direct built-in mechanism to explicitly mark tables as such in Delta Live Tables.
• However, you can adopt a practice to achieve this:
  • Landing Layer: Keep the original data (full history) in a separate “landing” layer. This layer remains untouched during refreshes.
  • Bronze/Silver/Gold Layers: The downstream layers (bronze, silver, gold) can be recalculated as needed, but the landing layer remains pristine.
  • By structuring your pipeline this way, you ensure that the original data remains intact while allowing for transformations and aggregations in subsequent layers.