cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

DLT Apply Changes

ankitmit
New Contributor II

Hi,

In DLT, how do we specify which columns we don't want to overwrite when using the “apply changes” operation in the DLT (in the attached example, we want to avoid overwriting the “created_time” column)?

I am using this sample code 

dlt.apply_changes(
source= f"{table_name}_cdc",
target= table_name,
keys = table_keys,
sequence_by = F.col('_fivetran_synced'),
except_column_list = ['created_time'],
stored_as_scd_type = 1,
apply_as_deletes=F.expr("_fivetran_deleted= true"),
track_history_column_list = None,
track_history_except_column_list = None
)

But instead, except_column_list drops this column from the target table.

3 REPLIES 3

filipniziol
Contributor

Hi @ankitmit ,

If you check the documentation of APPLY CHANGES, there is no parameter to exclude the column from being updated.

As you noticed, columns parameter is used to specify the columns included in the target table, so excluding created_table, means the column will not be available in the target table.




Finn-Ol
New Contributor II

There might be a misunderstanding regarding the except_column_list parameter in the apply_changes function. This parameter is used to specify which columns to exclude from the changes, but in this case, it seems like it’s dropping the created_time column instead of preserving it.

One approach to consider is ensuring that the data types and configurations are compatible with the intended functionality.

ankitmit
New Contributor II

Yes, that's exactly the problem. except_column_list is dropping the column from the target table instead of excluding them from the changes.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group