- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2025 09:27 AM
Hi, your understanding is accurate-with a few nuances about DLT and how it handles full refreshes with external source tables regarding deletes and retention.
- During full refresh : Clears the pipeline state and output tables , then reprocesses all data from source table as it currently exists.
- Handling deleted in the source table: Deletes in the source table are not automatically tracked unless you implement a Change Data Capture (CDC)
- By default, DLT streaming live tables operate in append-only mode and do not handle deletes or updates. Only new rows are processed.
-
If your source table has a retention policy but retains historical versions—such as with Delta tables—then a full refresh pulls in all data present in the current physical state of that source (meaning, whatever is visible at the time of the DLT refresh t = now). Deletes that are still present because of retention may appear in CDC if you use it, but not in ordinary append-only ingestion pipelines
-
If retention purges deleted data (via VACUUM), then deleted records truly disappear and won’t be visible in the full refresh either.
- Use CDC and /or row tracking to ensure deletes and updates in the source table are tracked and correctly reflected downstream through DLT.
- Retention of historical data in the source can cause more data to be reprocessed if those records persist in the source table at refresh time.
- Full refresh reprocesses all currently available data, which means it can indeed reprocess historical data (including anything not deleted at the source) within retention limits.