topic Re: Data contracts on Databricks: what does a working MVP actually look like? in Data Governance

Data contracts on Databricks: what does a working MVP actually look like?

binlogreader — Sat, 25 Jul 2026 20:03:46 GMT

Hi All, I'm trying to solve two related problems on our lakehouse and I keep arriving at data contracts as the answer, but I can't find much on what a first working version looks like in practice on Databricks:

(a) Upstream application teams change their database schemas without telling us. We ingest their tables through CDC, and schema evolution keeps the pipelines alive (new columns get added or land in the rescued data column), but that is absorption, not communication. We find out after the change is already flowing, and purely semantic changes, like a column's meaning or allowed values shifting, never trip anything at all.

(b) When a change reaches our silver and gold tables, consumers find out when their dashboards or queries break. Unity Catalog lineage tells them where data came from, but nothing tells them ahead of time that something is about to change.

Here is the the closest end-to-end example on building a data product with Databricks. A datacontract.yaml per data product, the Data Contract CLI to generate DDL and run acceptance tests against the actual Unity Catalog table, deployment through Asset Bundles, tests running as job tasks, and metadata published to a registry so consumers can discover the product. It's a good picture, but it's a demo of one product, and it leaves my main questions open.

While thinking about what an MVP may look like, here are some questions that I request some guidance on:

1. Where does the contract live, and who owns it? For the upstream problem, the contract only helps if the application team's CI fails when their migration breaks it. Has anyone actually gotten an application team to adopt a contract for tables they consider their internal database?
2. Where do you enforce? Options seem to be CI (datacontract test on a schedule or on deploy), runtime (DLT expectations generated from the contract), or the catalog itself (constraints, tags). What combination is worth the effort first
3. How do downstream consumers actually get notified? A registry is discovery, not notification. Is anyone driving alerts or a changelog from contract diffs, and how?
4. Is anyone running the Data Contract CLI against Unity Catalog in production? How does it coexist with schema evolution modes that are deliberately permissive at bronze?

Appreciate your time!

Re: Data contracts on Databricks: what does a working MVP actually look like?

emma_s — Tue, 28 Jul 2026 15:18:50 GMT

Hi,

You've hit a problem that most data teams experience, some more than others! From my personal experience I'd be very wary about going down the data contract route. Unless you can get true ownership from the application team, which I've not yet seen. Usually, I've seen that it ends up as a sticking point to ingesting new datasets and what you normally end up testing with them is whether the contract is right rather than whether the actual dataset is right. But to answer your questions.

1. Usually the contract is stored as Yaml, in the data teams github repo, it could be put in a UC volume. It could work to have it in the App dev team's repo, but it if you have your data engineering patterns in a different repo this could cause issues.

2. I've seen it deployed in both the CI or the runtime. Never on the catalog. I think this would be challenging as there aren't many things that can actually be enforce on the table itself.

3. Usually, you would build the alerts into your CI or Pipelines and define who you would notify and this goes back to 1, who owns the contract? Do you want to be in a position where the appdev team changes something it breaks your pipeline and then you're reliant on them to fix it, which may not be a top prioirty for them.

4. I'm not convinced the two things can coexist together. I think on a table, you either need data contracts or schema evolution modes. My advice is to consider the purpose of the contract. When we had to make this situation better, we actually just added the need for a data engineering reviewer on any PRs that required a schema change. A lot of this, though, will depend on the culture and nature of the organisation you work for.

I hope this helps you think about it. As a side note, these are personal experiences rather than sanctioned Databricks best practices.

Many Thanks,

Emma

Re: Data contracts on Databricks: what does a working MVP actually look like?

DoTA — Wed, 29 Jul 2026 13:51:44 GMT

In practice, on large enterprise scale I found that applying data contracts is very tricky, and there are some major problem to it:

1. There are too many legacy systems, and aligning the owners of the system and the ability to alter the system alone consume too much energy (if the system even allow changes without going to a vendor and begging them for a CR which cause thousand of dollars). And, there are not only internal systems but also external systems (such as GA firebase, appsflyer as an example) which does not provide data contract OOTB.

2. If you manage to figure out part 1, applying data contract is not always straight forward, it really depends on the consumer. Some consumers want to stop the pipeline as soon as there is something wrong, some doesnt. Hence the general approach is to apply a pre_check (this is a task in the workflow) where you define the condition to allow the task to continue its transformation process. But it all boiled down to alignment and enterprise design, do not couple alert and stopping mechanism into a single pipeline as you will find your self aligning among different people with different risk appetite. Seperate them out of the golden layer (if you are following medallion architecture), put the hard stop in the consumption layer.

3. If you manage to get here, the alerting is easy, put it in the pre-check of the workflow it self. But beware, you will have thousands of alert everyday, so better to consolidate into a metadata table and send that table as a report daily.

Re: Data contracts on Databricks: what does a working MVP actually look like?

binlogreader — Thu, 30 Jul 2026 02:48:45 GMT

@emma_s @DoTA Thank you for your incredibly detailed notes! I am going to sit with them for a while to internalize, and then will respond back with some ideas or questions.

Re: Data contracts on Databricks: what does a working MVP actually look like?

honey_sharma — Thu, 30 Jul 2026 06:05:29 GMT

A practical MVP is to keep Bronze permissive for ingestion, but enforce contracts from Silver onward. Store the contract in Git, version it alongside pipeline code, validate it in CI using the Data Contract CLI, and publish approved changes with a simple changelog. For notifications, integrate contract diffs with Slack, Teams, or email so downstream consumers know before deployment. Ownership should ideally stay with the data product owner, with upstream teams reviewing contract changes as part of their release process.