Hi All, I'm trying to solve two related problems on our lakehouse and I keep arriving at data contracts as the answer, but I can't find much on what a first working version looks like in practice on Databricks:
(a) Upstream application teams change their database schemas without telling us. We ingest their tables through CDC, and schema evolution keeps the pipelines alive (new columns get added or land in the rescued data column), but that is absorption, not communication. We find out after the change is already flowing, and purely semantic changes, like a column's meaning or allowed values shifting, never trip anything at all.
(b) When a change reaches our silver and gold tables, consumers find out when their dashboards or queries break. Unity Catalog lineage tells them where data came from, but nothing tells them ahead of time that something is about to change.
Here is the the closest end-to-end example on building a data product with Databricks. A datacontract.yaml per data product, the Data Contract CLI to generate DDL and run acceptance tests against the actual Unity Catalog table, deployment through Asset Bundles, tests running as job tasks, and metadata published to a registry so consumers can discover the product. It's a good picture, but it's a demo of one product, and it leaves my main questions open.
While thinking about what an MVP may look like, here are some questions that I request some guidance on:
1. Where does the contract live, and who owns it? For the upstream problem, the contract only helps if the application team's CI fails when their migration breaks it. Has anyone actually gotten an application team to adopt a contract for tables they consider their internal database?
2. Where do you enforce? Options seem to be CI (datacontract test on a schedule or on deploy), runtime (DLT expectations generated from the contract), or the catalog itself (constraints, tags). What combination is worth the effort first
3. How do downstream consumers actually get notified? A registry is discovery, not notification. Is anyone driving alerts or a changelog from contract diffs, and how?
4. Is anyone running the Data Contract CLI against Unity Catalog in production? How does it coexist with schema evolution modes that are deliberately permissive at bronze?
Appreciate your time!
