- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a week ago
The concept is supported — but there's one detail in your code that's likely the real risk, and it's not the full refresh itself.
1. Deletion detection: yes, that's literally what the API is designed for. AUTO CDC FROM SNAPSHOT (the new name for apply_changes_from_snapshot — same signature, Databricks recommends renaming) compares consecutive snapshots and infers inserts, updates and deletes; keys present in the target but gone from the source get deleted — which for SCD2 means the record is closed with an end date. So your weekly full refresh surfacing Salesforce deletions is exactly the use case.
2. Here's my concern: your code wraps the source in spark.readStream.table("raw.contact") and feeds THAT to the snapshot API. A streaming read on a table that gets fully rewritten once a week is where checkpoint pain lives — streaming reads error out when existing records change or disappear underneath them. The snapshot API doesn't want a stream at all: in periodic snapshot mode you pass the source as a plain table name (source="raw.contact"), and each pipeline update it reads the current state of that table and diffs it against the target. No streaming checkpoint on the source, so an upstream full refresh is just "the next snapshot" — which is the whole point. I'd drop the intermediate streaming table and pass the name directly. And you're right NOT to full-refresh the SCD2 pipeline — that would rebuild it from the current snapshot only and wipe your history.
3. Schema changes: this is the honest gap. New columns generally flow through, but I couldn't find documentation describing what happens when a column is REMOVED from the source for this specific API — so I won't guess. I'd test column removal in a dev pipeline before trusting it, or protect yourself with an explicit select of the columns you track so source drift doesn't surprise the target.
One small design note: since deletions only become visible at the weekly full refresh, your SCD2 end dates for deleted records will be accurate to the week, not the day. If that's fine for your consumers, great — just worth stating explicitly.
Principal Data & AI — CI&T
thomazn@ciandt.com
linkedin.com/in/thomaz-antonio-rossito-neto