Streaming read fails with "Detected a data update" after a restatement job touches the source

[Islam_hoti]

Hi everyone,

Looking for the right pattern here rather than a workaround.

Setup. DBR 15.4 LTS, Unity Catalog. A silver streaming table reads from a bronze Delta table with a normal streaming read. Bronze is append only in the ordinary course of business, so this has run cleanly for months.

The problem started when we introduced a monthly restatement job. Once a month, a correction process runs an UPDATE against bronze to fix a mis-mapped source code across historical rows. The next time the silver stream runs, it fails with the error about a data update being detected in the source table, and refuses to continue.

Compaction is not the issue. Regular OPTIMIZE runs have never broken this stream, which matches my understanding that compaction commits are marked as not changing data. It is specifically the restatement UPDATE that breaks it.

What I am weighing.

skipChangeCommits would let the stream continue, but it ignores the restated rows entirely, so silver would keep the wrong values forever. That seems worse than failing loudly.

Reading the change data feed instead of the table would let me propagate the corrections properly, but it changes the shape of the downstream logic, and I would need to handle update preimage and postimage records rather than plain appends.

Full refresh of silver after each restatement is the brute force option. It works, but silver is large enough that this is not something I want to do monthly.

Questions.

Is switching to a change data feed read the accepted answer here, or is there a pattern people prefer when corrections are rare and batched?

If I move to a CDF read, is there a clean way to cut over without reprocessing all of history, given silver is already correct up to a known point?

Does anyone restate through a separate correction stream instead, leaving the main append path untouched? That feels cleaner architecturally but I have not seen it written down anywhere.

Thanks.

[AbhilashNagilla]

On DBR 15.4 LTS, use legacy CDF when corrections must reach silver, provided CDF was enabled before this UPDATE and the required version remains retained (source changes, CDF availability and retention).

1. Before MERGE, reduce each microbatch to one final action per target key using _commit_version; multiple source rows for one target key can make MERGE fail on DBR 15.4. Use update_postimage for stable-key updates; if a key can change, also turn the preimage's old key into a delete (CDF schema, streaming MERGE, duplicate matches).

2. For a self-managed stream whose target reflects the source through exactly version V, start CDF at inclusive version V + 1 with a new checkpoint after confirming CDF covers V + 1 and that version remains in history (starting version).

3. For rare updates, pause the append stream, batch-read the exact inclusive CDF versions, apply the correction idempotently, then resume with skipChangeCommits only after it succeeds (separate logic, batch ranges, idempotent streaming MERGE). If CDF doesn't cover this UPDATE, rerun the existing silver transformation for the affected scope; use a direct keyed repair only if that transformation is stateless and produces one target row per stable key (Delta MERGE).

[Khasim_1 · Accepted solution]

 This is a classic "Streaming vs. Batch" impedance mismatch. You are encountering the "Data change detected" exception because standard Spark Structured Streaming (and by extension, the DLT/Lakeflow logic) assumes the source is an append-only event log. An UPDATE operation violates the monotonic contract the stream expects.

Here is the breakdown of the patterns you are weighing, and the industry-standard recommendation for this specific "monthly restatement" scenario.

1. Is Change Data Feed (CDF) the accepted pattern?

Yes. Moving to CDF is the most robust, "native" way to handle updates in a streaming pipeline. It explicitly tells the downstream consumer: "Here is a set of changes (pre-images and post-images)," which Structured Streaming is built to handle via readStream on the table with CDF enabled.

• The "Workaround" Trap: skipChangeCommits is a destructive pattern. As you correctly identified, ignoring updates means your "Silver" layer becomes a "partial truth" repository, which undermines the entire Medallion architecture.

1. Can you cut over to CDF without a full reprocess?

Yes, but it requires a "split-brain" deployment. You cannot simply point your existing Silver stream at a table that now has CDF enabled and expect it to "magically" pick up the updates. The streaming checkpoint offset is linked to the table version.

The "Clean Cut-over" Pattern:

1. Stop the existing Silver stream.
2. Snapshot the current Silver state.
3. Create a new Silver stream (or a new version of the existing pipeline) that reads from table_changes(bronze_table, version_start).
4. Set startingVersion to the version after your last successful stream processing.
5. Re-process only the delta: Since you are reading changes, you only process the updates from that point forward. Note: You may need to write a small "adapter" logic to transform the CDF pre/post images into an upsert/merge pattern in Silver.

1. The "Correction Stream" Pattern (The "Cleaner" Alternative)

You mentioned a "separate correction stream," and this is actually the preferred pattern for high-integrity financial or audit systems.

Instead of UPDATEing your Bronze table, use the "Delete-and-Re-insert" (or "Correction-Log") Pattern:

• Pattern: Do not update Bronze. Treat Bronze as an immutable ledger.
• Correction Table: Create a second "Correction" table where you land the fixed records.
• Union Strategy: In your Silver pipeline, perform a UNION ALL of the "Bronze Main" stream and the "Correction" stream.
• Merge in Silver: Use APPLY CHANGES INTO (if using Lakeflow/DLT) or a MERGE statement in Silver.
• The MERGE logic naturally handles the update by looking for the key.
• Since the Correction stream arrives later (or with a higher sequence number/timestamp), it naturally overwrites the Silver record.

Why this is better:

• Bronze remains immutable: You never violate the "Append-only" assumption of the Bronze stream.
• Auditability: You have a physical record of the "Correction" in a separate table, rather than masking historical source data with an UPDATE.