How should schema evolution be handled across silver and gold layers in a medallion architecture?

temarych
New Contributor

Monday

We run a medallion pipeline on Databricks:

  • Bronze: AutoLoader ingests raw CSV files into Delta tables (append-only, all columns as STRING, schema evolution via addNewColumns)
  • Silver: PySpark jobs clean and transform bronze data into Delta tables using batch writes and CDC merge
  • Gold: dbt incremental models aggregate silver into fact and measure tables

AutoLoader handles bronze schema evolution automatically. The problem is silver and gold.

When a source adds or renames a column, we currently handle it manually - authoring ALTER TABLE statements for silver Delta tables and running dbt run --full-refresh for gold. This works but is ad-hoc and error-prone, especially for column renames.

Our most frequent case is new columns being added by the source. We also care about dropped columns - we don't want dead columns accumulating in silver and gold tables over time.

Questions:

  1. Is there a recommended pattern for managing silver Delta table schema changes in a DAB-deployed pipeline - beyond manual ALTER TABLE?
  2. Should we introduce a schema migration tool like Alembic/Liquibase/Flyway for Delta tables?
Labels:
0 Kudos