Initial question broken down

• Bronze table is truncate+insert every second. How can I process this into Silver with DLT before data is wiped? Does DLT continuous mode have the power to keep up?
• If I instead use an append-only source, how do I stop it from growing forever (retention)?

1. Truncate+insert bronze problem

Streaming/DLT continuous assumes append-only. With truncate+insert you’ll always lose data (engine sees “reset” not “append”).

fix: don’t stream directly from bronze. Instead, capture each truncate-load as a full snapshot and append it into a new bronze_snapshots table. That preserves every second’s data before bronze is wiped.

That’s the bronze_snapshots DLT transform in the PoC.

2. Can DLT continuous handle it?

No, because the issue is not throughput but data semantics. Even if DLT is fast enough, it can’t checkpoint against a source that resets.

answer: run DLT in triggered batch mode (every second/few seconds). Each trigger captures the latest bronze snapshot and appends it. Then use APPLY CHANGES (or MERGE) to compute deltas into silver.

3. Append-only source growth / retention

Once you switch to bronze_snapshots (append-only), yes it will grow forever.

answer: control size with Delta retention + VACUUM + OPTIMIZE

1) Create an append-only snapshots table (DLT Python)

2) Apply changes from snapshots to silver (preferred: DLT apply_changes)

If your Databricks workspace supports DLT apply_changes() (APPLY CHANGES INTO), use it because it’s declarative and handles ordering and deletes.