cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT Apply Changes

ankitmit
New Contributor III

Hi,

In DLT, how do we specify which columns we don't want to overwrite when using the “apply changes” operation in the DLT (in the attached example, we want to avoid overwriting the “created_time” column)?

I am using this sample code 

dlt.apply_changes(
source= f"{table_name}_cdc",
target= table_name,
keys = table_keys,
sequence_by = F.col('_fivetran_synced'),
except_column_list = ['created_time'],
stored_as_scd_type = 1,
apply_as_deletes=F.expr("_fivetran_deleted= true"),
track_history_column_list = None,
track_history_except_column_list = None
)

But instead, except_column_list drops this column from the target table.

5 REPLIES 5

filipniziol
Esteemed Contributor

Hi @ankitmit ,

If you check the documentation of APPLY CHANGES, there is no parameter to exclude the column from being updated.

As you noticed, columns parameter is used to specify the columns included in the target table, so excluding created_table, means the column will not be available in the target table.




Finn-Ol
New Contributor II

There might be a misunderstanding regarding the except_column_list parameter in the apply_changes function. This parameter is used to specify which columns to exclude from the changes, but in this case, it seems like it’s dropping the created_time column instead of preserving it.

One approach to consider is ensuring that the data types and configurations are compatible with the intended functionality.

ankitmit
New Contributor III

Yes, that's exactly the problem. except_column_list is dropping the column from the target table instead of excluding them from the changes.

HoussemBL
New Contributor III

I am facing a similar issue. Is there any solution to select which columns to update when using "apply changes" operation?

Anish_2
New Contributor II

In my case, 3 files were loaded at same time which has records with same keys but increasing sequence by value. I added date column in track_history_except_column_list in order to avoid tracking of history but values of date column came different for all records. Can anyone explain why this is happening?

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now