NandiniN
Databricks Employee
Databricks Employee

When dealing with Change Data Capture (CDC) in Delta Live Tables, it's crucial to handle out-of-order data correctly. You can use the APPLY CHANGES API to manage this. The APPLY CHANGES API ensures that the most recent data is used by specifying a column in the source data to sequence records.

dlt.apply_changes(
    target="vendas_silver",
    source="vendas",
    keys=["venda_id"],
    sequence_by=col("dth_ingestao"),
    apply_as_deletes=expr("operation = 'DELETE'"),
    except_column_list=["operation", "dth_ingestao"],
    stored_as_scd_type="1"
)

Which you are doing already, now ensure that you create and refresh the streaming tables correctly. This will help in propagating the updates downstream.

CREATE OR REFRESH STREAMING TABLE vendas_silver;
APPLY CHANGES INTO live.vendas_silver
FROM stream(vendas_raw)
KEYS (venda_id)
APPLY AS DELETE WHEN operation = 'DELETE'
SEQUENCE BY dth_ingestao
COLUMNS * EXCEPT (operation, dth_ingestao)
STORED AS SCD TYPE 1;

 

Read more here - https://docs.databricks.com/en/delta-live-tables/cdc.html