Alberto_Umana
Databricks Employee
Databricks Employee

Hi @Fatimah-Tariq,

Thanks for your question. 

When a schema change involves the data type of a column, and this column is part of a primary key, Delta Lake's schema evolution mechanism can handle it by updating the schema. But, this change can sometimes inadvertently cause data inconsistencies, such as setting previous records to null if the old data cannot be cast perfectly into the new data type.

Use MERGE INTO with the AUTO-OPTIMIZE and AUTO-COMPACTION features to ensure smooth transitions between schema versions.

I will check internally what could be best option to address this , or if a alter to come back to previous data type should be the way.