Schema Evolution/Type Widening in Materialized Views

[ismaelhenzel]

My team is migrating pipelines from Spark to Delta Live Tables (DLT), but we've found that some important features, like schema evolution for tables with enforced schemas, seem to be missing. In DLT, we can define schemas, set primary and foreign keys, add comments, and apply constraints. However, I can't find a way to manage changes to an enforced schema. For example, how can I update a column's data type from DECIMAL(10,5) to DECIMAL(11,5) without breaking the pipeline or having to fully reload the table? 

(i don't want to loose the schema of my mv table)

[nayan_wylde]

DLT supports schema evolution, but changing column data types (like from DECIMAL(10,5) to DECIMAL(11,5)) is not automatically handled. Here's how you can manage it:

Option 1: Full Refresh with Schema Update

If you're okay with refreshing the materialized view (MV) or target table:

1. Update the schema in your DLT code (e.g., change the column type in the StructType definition).
2. Use a helper function to process _rescued_data and cast it to the new type.
3. Trigger a full refresh of the affected table (e.g., silver layer) to apply the new schema.

This method keeps your MV intact and avoids reloading the bronze layer.

Option 2: Append-Only Strategy with Union View

If you want to avoid full refreshes:

1. Create a new version of the table (e.g., silver_user_v1) with the updated schema.
2. Use @Dlt.append_flow to append data from both old and new sources.
3. Create a view that unions the old and new tables.