cdc type 2 - not closing records

MariuszJ — Wed, 06 Sep 2023 11:25:39 GMT

Hi,
i would not mind small advice, i do have dlt cdc typ2, the definition

dlt.create_streaming_table('`my_table_dlt_cdc`') dlt.apply_changes( target = 'my_table_dlt_cdc', source = 'source', keys = ['id'], sequence_by = col('snapshot_date'), # ignore_null_updates = True, except_column_list = ['snapshot_date'], stored_as_scd_type = "2" )

the table it self is very simple

id| val | snapshot_date 1 | 'abc' | 2023-01-01

The '1' is stopped to exist in source table after march, but cdc never closed the row. Data for whole 2023 has been loaded (up to today).

When querying __apply_changes_storage_my_table_dlt_cdc i do see last entry in march but nothing after, row was not closed.

id,value,__recordStartAt,__START_AT,__END_AT,__rowIsHidden 1,'ABC',2023-03-17,2019-07-05,null,true 1,'ABC',2023-03-24,2019-07-05,null,true 1,'ABC',2023-03-31,2019-07-05,null,false

I do have a feeling that I'm missing something obvious... But what.

Re: cdc type 2 - not closing records

MariuszJ — Wed, 13 Sep 2023 07:48:46 GMT

For anyone who sees this post in the future. I was missing one argument

apply_as_deletes

topic Re: cdc type 2 - not closing records in Data Engineering

cdc type 2 - not closing records

Re: cdc type 2 - not closing records