Louis_Frolio
Databricks Employee
Databricks Employee

 

Hi @maikel ,

Good question, and a pretty common pain point once you start promoting to higher environments.

Databricks doesn't ship a dedicated schema migration framework, so the standard approach is what you'd expect: keep your DDL and seed SQL in version control, automate execution in order per environment, and write scripts to be idempotent so they're safe to re-run. You don't need to move to Python just to manage this well.

For environment layout, separate catalogs or schemas per environment in Unity Catalog with consistent naming works well — e.g. dev.analytics.orders, test.analytics.orders, prod.analytics.orders.

Store migrations as ordered files:

  • 001_create_base_schemas.sql
  • 002_create_orders_tables.sql
  • 003_seed_reference_data.sql

Write them idempotent — CREATE TABLE IF NOT EXISTS, MERGE for seed data instead of plain INSERT, ALTER TABLE for structural changes where you can.

Track what's run with a simple migration history table in each environment:

CREATE TABLE IF NOT EXISTS admin.schema_migrations (
  version STRING,
  applied_at TIMESTAMP
);

Your deployment job reads which versions are already applied, runs only the new files, and logs a row after each successful migration. Nothing fancy, but it works.

For the dev → test → prod promotion side, Databricks Asset Bundles (DABs) is worth a look. It's not a schema migration tool on its own, but it handles promoting jobs and pipelines across environments with variable overrides per environment — pairs naturally with the versioned SQL pattern above.

On Alembic: it works, and you can keep migrations as mostly raw SQL so you retain full control over Delta DDL. Makes sense if your team is already standardized on it elsewhere. If not, the SQL-only approach is simpler and a lot more accessible to folks who aren't deep in Python.

Bottom line — you don't need Alembic to do this well. Version-controlled SQL + a migration history table + automated execution per environment is a solid, widely-used pattern. Happy to dig into any piece of this further.

Cheers, Lou