Use dynamic schema handling and selective table updates to apply metadata changes incrementally from the current watermark, preserving history across country go-lives.
Replace static @dlt.table definitions with Auto Loader's schema inference/evolution—handles column adds/drops/types without code changes
Metadata Change Workflow (No Full Refresh)
- Update silver/gold logic (add/drop columns, new CTEs).
- Start pipeline update → DLT only refreshes affected tables downstream.
- Bronze Auto Loader resumes incrementally using existing schema location + checkpoints
New country? Update isin() filter → incremental only.
Validate before Go Live:
This eliminates 7-hour refreshes—new metadata propagates in ~10-30 minutes depending on daily delta volume. Your continuous mode stays live during changes. Test on one country first?
