DLT Apply Changes

[ankitmit]

Hi,

In DLT, how do we specify which columns we don't want to overwrite when using the “apply changes” operation in the DLT (in the attached example, we want to avoid overwriting the “created_time” column)?

I am using this sample code 

dlt.apply_changes(

source= f"{table_name}_cdc",

target= table_name,

keys = table_keys,

sequence_by = F.col('_fivetran_synced'),

except_column_list = ['created_time'],

stored_as_scd_type = 1,

apply_as_deletes=F.expr("_fivetran_deleted= true"),

track_history_column_list = None,

track_history_except_column_list = None

)

But instead, except_column_list drops this column from the target table.

[filipniziol]

Hi @ankitmit ,

If you check the documentation of APPLY CHANGES, there is no parameter to exclude the column from being updated.

As you noticed, columns parameter is used to specify the columns included in the target table, so excluding created_table, means the column will not be available in the target table.

[Finn-Ol]

There might be a misunderstanding regarding the except_column_list parameter in the apply_changes function. This parameter is used to specify which columns to exclude from the changes, but in this case, it seems like it’s dropping the created_time column instead of preserving it.

One approach to consider is ensuring that the data types and configurations are compatible with the intended functionality.

[ankitmit]

Yes, that's exactly the problem. except_column_list is dropping the column from the target table instead of excluding them from the changes.

[HoussemBL]

I am facing a similar issue. Is there any solution to select which columns to update when using "apply changes" operation?

[Anish_2]

In my case, 3 files were loaded at same time which has records with same keys but increasing sequence by value. I added date column in track_history_except_column_list in order to avoid tracking of history but values of date column came different for all records. Can anyone explain why this is happening?

[juice]

I'm experience same issue where the columns added to except_column_list are being dropped instead of ignored.

[brunoillipronti]

Same here, it's kinda ridiculous that apply_changes doesn't support a parameter to update certain columns... how come that is not a priority since this was released?