Auto CDC Delete Propagation Issue: Streaming CDF Reads Don't Capture Delete Events from Auto CDC

Surya2
New Contributor III

a month ago

Summary

I'm exploring GDPR delete propagation through a medallion architecture (Bronze → Silver → Gold) using Auto CDC with Change Data Feed. Delete events propagate successfully from Landing → Bronze, but fail to propagate from Bronze → Silver → Gold, even though deletes exist in Bronze's CDF.

Architecture

Landing (regular Delta + CDF enabled)
  → Bronze (Auto CDC + CDF enabled)
    → Silver (Auto CDC + CDF enabled)
      → Gold (Auto CDC target)

What Works

  • ✅ Landing → Bronze: Delete events propagate correctly
  • ✅ Bronze CDF contains delete events (verified via table_changes())
  • ✅ Update events propagate through all layers

What Doesn't Work

  • ❌ Bronze → Silver: Delete events are NOT captured when Silver reads Bronze's CDF
  • ❌ Silver → Gold: Delete events are NOT captured when Gold reads Silver's CDF

Question

Is chaining Auto CDC flows via streaming CDF reads a supported pattern for propagating deletes?

Specifically: When an Auto CDC target table has CDF enabled, and a downstream layer reads that table's CDF using .option("readChangeFeed", "true"), do streaming reads emit delete events?

Labels:
0 Kudos