paolajara
Databricks Employee
Databricks Employee

Hi, your understanding is accurate-with a few nuances about DLT  and how it handles full refreshes with external source tables regarding deletes and retention.

  1. During full refresh : Clears the pipeline state and output tables , then reprocesses all data from source table as it currently exists. 
  2. Handling deleted in the source table: Deletes in the source table are not automatically tracked unless you implement a Change Data Capture (CDC)
    • By default, DLT streaming live tables operate in append-only mode and do not handle deletes or updates. Only new rows are processed.
  3. If your source table has a retention policy but retains historical versions—such as with Delta tables—then a full refresh pulls in all data present in the current physical state of that source (meaning, whatever is visible at the time of the DLT refresh t = now). Deletes that are still present because of retention may appear in CDC if you use it, but not in ordinary append-only ingestion pipelines

  4. If retention purges deleted data (via VACUUM), then deleted records truly disappear and won’t be visible in the full refresh either.

  5. Use CDC and /or row tracking to ensure deletes and updates in the source table are tracked and correctly reflected downstream through DLT. 
  6. Retention of historical data in the source can cause more data to be reprocessed if those records persist in the source table at refresh time.
  7. Full refresh reprocesses all currently available data, which means it can indeed reprocess historical data (including anything not deleted at the source) within retention limits.

View solution in original post