Schema Evolution/Type Widening in Materialized Views
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-25-2025 11:09 AM
My team is migrating pipelines from Spark to Delta Live Tables (DLT), but we've found that some important features, like schema evolution for tables with enforced schemas, seem to be missing. In DLT, we can define schemas, set primary and foreign keys, add comments, and apply constraints. However, I can't find a way to manage changes to an enforced schema. For example, how can I update a column's data type from DECIMAL(10,5) to DECIMAL(11,5) without breaking the pipeline or having to fully reload the table?
(i don't want to loose the schema of my mv table)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-25-2025 12:52 PM - edited 08-25-2025 12:53 PM
DLT supports schema evolution, but changing column data types (like from DECIMAL(10,5) to DECIMAL(11,5)) is not automatically handled. Here's how you can manage it:
Option 1: Full Refresh with Schema Update
If you're okay with refreshing the materialized view (MV) or target table:
- Update the schema in your DLT code (e.g., change the column type in the StructType definition).
- Use a helper function to process _rescued_data and cast it to the new type.
- Trigger a full refresh of the affected table (e.g., silver layer) to apply the new schema.
This method keeps your MV intact and avoids reloading the bronze layer.
Option 2: Append-Only Strategy with Union View
If you want to avoid full refreshes:
- Create a new version of the table (e.g., silver_user_v1) with the updated schema.
- Use @Dlt.append_flow to append data from both old and new sources.
- Create a view that unions the old and new tables.