Schema Evolution/Type Widening in Materialized Views

ismaelhenzel — Mon, 25 Aug 2025 18:09:44 GMT

My team is migrating pipelines from Spark to Delta Live Tables (DLT), but we've found that some important features, like schema evolution for tables with enforced schemas, seem to be missing. In DLT, we can define schemas, set primary and foreign keys, add comments, and apply constraints. However, I can't find a way to manage changes to an enforced schema. For example, how can I update a column's data type from DECIMAL(10,5) to DECIMAL(11,5) without breaking the pipeline or having to fully reload the table?

(i don't want to loose the schema of my mv table)

Re: Schema Evolution/Type Widening in Materialized Views

nayan_wylde — Mon, 25 Aug 2025 19:53:16 GMT

DLT supports schema evolution, but changing column data types (like from DECIMAL(10,5) to DECIMAL(11,5)) is not automatically handled. Here's how you can manage it:

Option 1: Full Refresh with Schema Update

If you're okay with refreshing the materialized view (MV) or target table:

Update the schema in your DLT code (e.g., change the column type in the StructType definition).
Use a helper function to process _rescued_data and cast it to the new type.
Trigger a full refresh of the affected table (e.g., silver layer) to apply the new schema.

This method keeps your MV intact and avoids reloading the bronze layer.

Option 2: Append-Only Strategy with Union View

If you want to avoid full refreshes:

Create a new version of the table (e.g., silver_user_v1) with the updated schema.
Use @Dlt.append_flow to append data from both old and new sources.
Create a view that unions the old and new tables.

topic Re: Schema Evolution/Type Widening in Materialized Views in Data Engineering

Schema Evolution/Type Widening in Materialized Views

Re: Schema Evolution/Type Widening in Materialized Views

Option 1: Full Refresh with Schema Update

Option 2: Append-Only Strategy with Union View