topic Re: DLT Apply Changes in Data Engineering

DLT Apply Changes

ankitmit — Tue, 15 Oct 2024 14:05:01 GMT

Hi,

In DLT, how do we specify which columns we don't want to overwrite when using the “apply changes” operation in the DLT (in the attached example, we want to avoid overwriting the “created_time” column)?

I am using this sample code

dlt.apply_changes(

source= f"{table_name}_cdc",

target= table_name,

keys = table_keys,

sequence_by = F.col('_fivetran_synced'),

except_column_list = ['created_time'],

stored_as_scd_type = 1,

apply_as_deletes=F.expr("_fivetran_deleted= true"),

track_history_column_list = None,

track_history_except_column_list = None

)

But instead, except_column_list drops this column from the target table.

Re: DLT Apply Changes

filipniziol — Tue, 15 Oct 2024 16:28:02 GMT

Hi @ankitmit ,

If you check the documentation of APPLY CHANGES, there is no parameter to exclude the column from being updated.

As you noticed, columns parameter is used to specify the columns included in the target table, so excluding created_table, means the column will not be available in the target table.

Re: DLT Apply Changes

Finn-Ol — Tue, 15 Oct 2024 16:29:00 GMT

There might be a misunderstanding regarding the except_column_list parameter in the apply_changes function. This parameter is used to specify which columns to exclude from the changes, but in this case, it seems like it’s dropping the created_time column instead of preserving it.

One approach to consider is ensuring that the data types and configurations are compatible with the intended functionality.

Re: DLT Apply Changes

ankitmit — Wed, 23 Oct 2024 06:53:14 GMT

Yes, that's exactly the problem. except_column_list is dropping the column from the target table instead of excluding them from the changes.

Re: DLT Apply Changes

HoussemBL — Wed, 06 Nov 2024 08:50:46 GMT

I am facing a similar issue. Is there any solution to select which columns to update when using "apply changes" operation?

Re: DLT Apply Changes

Anish_2 — Thu, 30 Jan 2025 19:04:39 GMT

In my case, 3 files were loaded at same time which has records with same keys but increasing sequence by value. I added date column in track_history_except_column_list in order to avoid tracking of history but values of date column came different for all records. Can anyone explain why this is happening?

Re: DLT Apply Changes

juice — Fri, 09 May 2025 01:09:27 GMT

I'm experience same issue where the columns added to except_column_list are being dropped instead of ignored.

Re: DLT Apply Changes

brunoillipronti — Tue, 24 Jun 2025 23:48:58 GMT

Same here, it's kinda ridiculous that apply_changes doesn't support a parameter to update certain columns... how come that is not a priority since this was released?