Change Data Feed on Materialized Views Why I Think This Is More Than an Incremental Processing

[AmitDECopilot]

One interesting way to look at CDF on Materialized Views is the difference between source-level change and business-level change.

CDC might tell us that five banking transactions changed. But after those transactions pass through our transformations, what a downstream consumer may actually care about is:

Risk Score: 42 → 67
Monthly Spend: $8,500 → $11,200

This raises an interesting architectural question: can our data products tell consumers not only their current state, but what changed since the last processing cycle?

I explored this using a banking Customer 360 example, including potential patterns for downstream processing and reconciliation, as well as why CDF should not be treated as a replacement for persistent audit history.

Full article: From Materialized Views to Change-Aware Data Products.

[Phani_sannala]

Good point, @AmitDECopilot  that gap between "row changed" and "value actually changed enough to matter" is where most CDC setups trip up. We hit the same thing on a Gold-layer project: CDF told us data moved, but teams really wanted to know "did the number that matters cross a line" - like a risk score jumping into a new bucket. So we added a simple check that only alerts when that meaningful shift happens, not every time a row is touched.

One thing I'm curious about - when a source record comes in late or gets corrected, do you reprocess everything from raw again, or just patch the already-computed table directly?

[AmitDECopilot]

Great question. I’d generally avoid patching the already-computed Gold table directly.

For late-arriving or corrected records, I prefer replaying from the earliest reliable layer where the correction exists, but only for the impacted keys/time window rather than rebuilding everything from raw.

The important part is making the downstream logic idempotent. If the corrected source value changes the business outcome, the Gold layer should be recomputed for that affected grain and CDF can then expose the resulting change.

That also keeps lineage intact  otherwise direct Gold patches can become very difficult to explain later.

Your risk-score example is exactly the distinction I was trying to highlight: a technical change is not necessarily a meaningful business change