Hi,
i would not mind small advice, i do have dlt cdc typ2, the definition
dlt.create_streaming_table('`my_table_dlt_cdc`')
dlt.apply_changes(
target = 'my_table_dlt_cdc',
source = 'source',
keys = ['id'],
sequence_by = col('snapshot_date'),
# ignore_null_updates = True,
except_column_list = ['snapshot_date'],
stored_as_scd_type = "2"
)
the table it self is very simple
id| val | snapshot_date
1 | 'abc' | 2023-01-01
The '1' is stopped to exist in source table after march, but cdc never closed the row. Data for whole 2023 has been loaded (up to today).
When querying __apply_changes_storage_my_table_dlt_cdc i do see last entry in march but nothing after, row was not closed.
id,value,__recordStartAt,__START_AT,__END_AT,__rowIsHidden
1,'ABC',2023-03-17,2019-07-05,null,true
1,'ABC',2023-03-24,2019-07-05,null,true
1,'ABC',2023-03-31,2019-07-05,null,false
I do have a feeling that I'm missing something obvious... But what.
