Hi @Surya2 ,

Nice write-up. The symptom you're describing, where updates propagate cleanly but deletes quietly disappear, is a common one, and the good news is that the pattern you're after is fully supported. The break is almost certainly in how your downstream layer consumes the upstream change feed, not in how AUTO CDC handles deletes. Let me walk through it.

The short version is yes. Chaining AUTO CDC flows by having a downstream layer read the upstream target's change data feed via .option("readChangeFeed", "true") is a supported pattern, and streaming CDF reads do emit delete events. The docs are explicit that you can read a change data feed from a streaming table that is the target of AUTO CDC (or AUTO CDC FROM SNAPSHOT) the same way you'd read the feed from any other Delta table.

Two things are worth confirming before anything else. Your upstream AUTO CDC target needs to be a Unity Catalog streaming table, and you need to be on Databricks Runtime 15.2 or above. If either of those isn't true, that's the first thing to fix.

Now to the heart of it. Because updates propagate and only deletes fail, the cause is one of two things, and a single check tells them apart. Inspect your downstream (Silver) source view and confirm whether it actually carries the CDF metadata columns: _change_type, _commit_version, and _commit_timestamp.

If those columns are present, you're in the first case. Silver is reading the change feed correctly, but the AUTO CDC flow isn't translating _change_type = 'delete' rows into actual deletes. A delete event arrives carrying the row's values, and without an apply_as_deletes condition, AUTO CDC treats it as an ordinary upsert and effectively re-inserts the row. Update events (update_postimage) upsert correctly, which is exactly why your updates work and your deletes don't. The fix is to add the apply_as_deletes condition, shown below.

If those columns are missing, you're in the second case. The readChangeFeed option isn't being honored, and the read has fallen back to a normal, append-oriented streaming read, so delete events never reach Silver at all. Make sure the option is set, make sure you aren't using skipChangeCommits (it explicitly ignores source updates and deletes), and verify the columns then appear. There's also an internal pipeline setting some teams have reached for when reader options get dropped (pipelines.streamingFlowReadOptionsEnabled=true), but that one isn't in the public docs, so treat it as a last resort rather than a documented fix.

Here's the pattern that resolves both cases:

from pyspark import pipelines as dp
from pyspark.sql.functions import col, expr

@dp.temporary_view()
def bronze_cdf():
    return (
        spark.readStream
             .option("readChangeFeed", "true")             # second case: must be honored
             .table("catalog.schema.bronze_table")
             .where("_change_type != 'update_preimage'")   # drop preimages so they don't resurrect rows
    )

dp.create_streaming_table("silver_table")

dp.create_auto_cdc_flow(
    target             = "silver_table",
    source             = "bronze_cdf",
    keys               = ["id"],
    sequence_by        = col("_commit_version"),            # order out-of-order changes (or _commit_timestamp)
    apply_as_deletes   = expr("_change_type = 'delete'"),   # first case: the usual missing piece
    except_column_list = ["_change_type", "_commit_version", "_commit_timestamp"],
    stored_as_scd_type = 1,                                 # Type 1 = row physically removed (needed for GDPR)
)

The equivalent key clauses in SQL are APPLY AS DELETE WHEN _change_type = 'delete', SEQUENCE BY _commit_version, and STORED AS SCD TYPE 1. Apply the same shape again for Silver to Gold.

A few things to verify as you go. Confirm DBR 15.2+ and Unity Catalog for every intermediate AUTO CDC target. You already have CDF enabled on each layer, just keep in mind that CDF is retained only for a window, so very old deletes can age out. Use SCD Type 1, not Type 2, since Type 2 only marks the row expired (a tombstone) while GDPR removal needs Type 1's physical delete. Verify the CDF metadata columns actually show up in each downstream source view, since that's the fastest way to know which case you're in. And never use skipChangeCommits on these reads, because it silently drops the very deletes you're trying to propagate.

So the pattern works. Since your deletes are visible in Bronze's CDF but aren't reaching Silver, the issue is in how Silver consumes Bronze. Either it isn't reading the CDF as CDF (second case), or it's reading it but not mapping _change_type = 'delete' into apply_as_deletes (first case). Once that's corrected, deletes will propagate through Silver and Gold the same way your updates already do.

Regards, Louis.

References

1. Advanced AUTO CDC topics, reading CDF from AUTO CDC targets. https://docs.databricks.com/aws/en/ldp/cdc-advanced
2. The AUTO CDC APIs: Simplify change data capture with pipelines. https://docs.databricks.com/aws/en/ldp/cdc
3. create_auto_cdc_flow (Python reference). https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-apply-changes
4. Use Delta Lake change data feed on Databricks. https://docs.databricks.com/aws/en/delta/delta-change-data-feed
5. Propagating Deletes: Managing Data Removal using Delta Live Tables (Community technical blog). https://community.databricks.com/t5/technical-blog/propagating-deletes-managing-data-removal-using-d...