nayan_wylde
Esteemed Contributor II

DLT supports schema evolution, but changing column data types (like from DECIMAL(10,5) to DECIMAL(11,5)) is not automatically handled. Here's how you can manage it:

Option 1: Full Refresh with Schema Update

If you're okay with refreshing the materialized view (MV) or target table:

  1. Update the schema in your DLT code (e.g., change the column type in the StructType definition).
  2. Use a helper function to process _rescued_data and cast it to the new type.
  3. Trigger a full refresh of the affected table (e.g., silver layer) to apply the new schema.

This method keeps your MV intact and avoids reloading the bronze layer.

Option 2: Append-Only Strategy with Union View

If you want to avoid full refreshes:

  1. Create a new version of the table (e.g., silver_user_v1) with the updated schema.
  2. Use @Dlt.append_flow to append data from both old and new sources.
  3. Create a view that unions the old and new tables.