I have made a few errors in my previous response. Here's the corrected version.
What's documented about the ordering behavior
For SCD Type 1 targets, you can combine one AUTO CDC FROM SNAPSHOT flow with one or more AUTO CDC flows. The snapshot versi...
Hi. Please find the another update on the response below:Use @DP.materialized_view for the overwrite table and an append-only streaming table (@DP.table over an Auto Loader read) for the history table. I also need to retract something from my last r...
Skip the custom app and Lakebase sync. On classic compute, compute policies with pinned libraries, scoped per team or persona, cover this at the platform level, and they don't depend on serverless.
Why not a single init script
Databricks recommends ...
Short version: your Table1 is really a materialized view (batch read, fully recomputed each update), and Table2 needs to be a streaming table fed by an incremental source, not by Table1. The fix is to change how you ingest the CSV for the history ta...
The cleanest fix I know is to stop treating this as two overlapping SCD2 loads. Make the two flows idempotent against each other by giving them a shared ordering domain, so a full refresh can replay both without producing duplicates.
Why you get dupl...
