APPLY_CHANGES late arriving data

Gilg — Sat, 07 Oct 2023 20:13:32 GMT

Hi Team,

I have a DLT pipeline that uses APPLY_CHANGES to our Silver tables.

I am using Id as keys and timestamp to know the sequence of the incoming data.

Question: How does APPLY_CHANGES handles late arriving data?

i.e., for silver_table_1, the data comes for Id 1 at 08/10/2023 at 10:30pm. This record got inserted to silver_table_1. Next day, same Id 1 data arrives but the timestamp is 07/10/2023 at 08:00pm. As you can see the timestamp is different and is less than from the inserted record.

Cheers,

Re: APPLY_CHANGES late arriving data

Gilg — Tue, 10 Oct 2023 23:58:50 GMT

Hi Kaniz,

Will this work in SCD Type1 table? As I understand the __START_AT and __END_AT columns are only available in SCD Type 2 APPLY_CHANGES. When I do a quick select of the table using SQL Warehouse I do not see these columns in a Type 1 table.

Cheers,

topic APPLY_CHANGES late arriving data in Data Engineering

APPLY_CHANGES late arriving data

Re: APPLY_CHANGES late arriving data