Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-25-2025 12:52 PM - edited 08-25-2025 12:53 PM
DLT supports schema evolution, but changing column data types (like from DECIMAL(10,5) to DECIMAL(11,5)) is not automatically handled. Here's how you can manage it:
Option 1: Full Refresh with Schema Update
If you're okay with refreshing the materialized view (MV) or target table:
- Update the schema in your DLT code (e.g., change the column type in the StructType definition).
- Use a helper function to process _rescued_data and cast it to the new type.
- Trigger a full refresh of the affected table (e.g., silver layer) to apply the new schema.
This method keeps your MV intact and avoids reloading the bronze layer.
Option 2: Append-Only Strategy with Union View
If you want to avoid full refreshes:
- Create a new version of the table (e.g., silver_user_v1) with the updated schema.
- Use @Dlt.append_flow to append data from both old and new sources.
- Create a view that unions the old and new tables.