Hi,
I have a DLT pipeline that applies changes from a source table (cdctest_cdc_enriched) to a target table (cdctest), by the following code:
dlt.apply_changes(
target = "cdctest",
source = "cdctest_cdc_enriched",
keys = ["ID"],
sequence_by = col("tran_begin_time"),
apply_as_deletes = expr("operation = 1"),
except_column_list = ["operation","tran_begin_time"],
stored_as_scd_type = 1
)
Now if I add an 'update' row to the source table and update the pipeline, the update is not applied to the target table, even though it is added to the source table. The source table follows the structure of a cdc table in SQL Server.
A screenshot of the data that is in the source table is attached, as well as a screenshot of the target table not showing updated data (don't be fooled by some of the - updated values in test1 column, those are the result of a backfill).
Is this is a known issue? Has anyone else had trouble with applying update operations sql server style (where update creates 2 records, one with the data before update (operation 3). and one with the data after update (operation 4)? Or am I doing something completely wrong?
