Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-26-2025 04:18 AM
When dealing with Change Data Capture (CDC) in Delta Live Tables, it's crucial to handle out-of-order data correctly. You can use the APPLY CHANGES API to manage this. The APPLY CHANGES API ensures that the most recent data is used by specifying a column in the source data to sequence records.
dlt.apply_changes(
target="vendas_silver",
source="vendas",
keys=["venda_id"],
sequence_by=col("dth_ingestao"),
apply_as_deletes=expr("operation = 'DELETE'"),
except_column_list=["operation", "dth_ingestao"],
stored_as_scd_type="1"
)
Which you are doing already, now ensure that you create and refresh the streaming tables correctly. This will help in propagating the updates downstream.
CREATE OR REFRESH STREAMING TABLE vendas_silver;
APPLY CHANGES INTO live.vendas_silver
FROM stream(vendas_raw)
KEYS (venda_id)
APPLY AS DELETE WHEN operation = 'DELETE'
SEQUENCE BY dth_ingestao
COLUMNS * EXCEPT (operation, dth_ingestao)
STORED AS SCD TYPE 1;
Read more here - https://docs.databricks.com/en/delta-live-tables/cdc.html