Implementing a "Zero-Bus" Architecture with Unity Catalog

[Khasim_1]

Hi everyone,

I am currently refining the architecture for an end-to-end Lakehouse project and moving toward a "Zero-Bus" (Bus Architecture) model. My primary objective is to enforce dimension conformity (Customer, Product, Geography) across multiple independent Gold-layer data marts to ensure a single version of truth.

I would love to gather insights from the community on how you are implementing this at scale:

1. Dimension Conformation Strategy

• Layering: Are you consolidating dimensions into a "Master Silver" layer before they reach the data marts, or are you creating "Conformed Shared Dimensions" that all downstream Gold pipelines reference?
• Golden Record: What is your preferred pattern for maintaining these master records while handling high-volume SCD Type 2 updates?

1. Governance via Unity Catalog

• Physical Structure: How are you utilizing Unity Catalog to support this? Are you grouping conformed dimensions into a dedicated Catalog or Schema to provide clear ownership and access control?
• Access Control: How do you handle the trade-off between strict governance of shared dimensions and the need for decentralized teams to innovate within their own Gold marts?

1. Pipeline Decoupling & Schema Evolution

• Impact Analysis: How do you manage schema evolution on these shared dimensions without creating "cascading failures" in the downstream Gold marts that depend on them?
• Validation: Are you implementing any automated "Contract Testing" between the conformed dimension provider and the downstream consumers?

I am trying to balance the rigidity required for a "Zero-Bus" model with the inherent agility of the Medallion architecture.

How is your team handling this balance, and what lessons have you learned when scaling this across multiple business domains?

[DoTA]

Hi @Khasim_1, here is the pattern I would use for conformed dimensions on Unity Catalog, taking your questions in order.

1) Layering. I would avoid a "Master Silver" that everything funnels through, because it tends to become a bottleneck team. Treat conformed dimensions as a published data product with a named owner. Domain Silver stays decentralized, the conformed dimension is produced by one pipeline, and every Gold mart consumes it read-only.

2) Golden record plus SCD2 at volume. Keep the golden record (match/merge, survivorship rules) separate from history. Maintain a current-state dimension with a stable surrogate key, and write the SCD2 history as its own table with AUTO CDC / APPLY CHANGES ... STORED AS SCD TYPE 2. Marts then join on the surrogate key plus a validity window. The surrogate key must never change when attributes change, otherwise every downstream mart breaks at once.

3) Unity Catalog structure. A dedicated catalog (or at least a schema) for conformed dimensions works well, owned by a platform or data governance group. Grant consumers SELECT only and give the owning group MANAGE. Mart teams get full control in their own catalogs, so they can innovate without touching the shared layer. That is the cleanest answer to governance versus agility: strict where it is shared, free where it is local.

4) Schema evolution without cascading failures. Version the contract, not the table. Additive changes (new nullable columns) are low risk. Breaking changes (rename, type change, dropped column) go out as a new view version (dim_customer_v2) while v1 stays for a deprecation window. Marts should read through views rather than the base table so you can shim old shapes. Use lineage (system.access.table_lineage) to see which consumers are still on v1 before you retire it.

5) Contract testing. Worth it. Put expectations on the producer side (not null and unique on the surrogate key, no overlapping validity windows, row count within a tolerance of the previous run), and have consumers run a small schema check in CI. If the producer fails its expectations it should not publish the new version, so marts keep reading the last good one.

One thing I would settle before any tooling: who owns Customer, and what the change process is. The Unity Catalog setup follows naturally once that is agreed.

1) Layering. I would not build a "Master Silver" that everything funnels through. It becomes a bottleneck team within a quarter. Treat conformed dimensions as a published data product with a named owner. Domain Silver stays decentralized, and the conformed dimension is produced by one pipeline and consumed read-only by every Gold mart.

2) Golden record plus SCD2 at volume. Keep the golden record (match/merge, survivorship rules) separate from history. Maintain a current-state dim with a stable surrogate key, and write the SCD2 history as its own table with AUTO CDC / APPLY CHANGES ... STORED AS SCD TYPE 2. Marts then join on the surrogate key plus a validity window. The key point is that the surrogate key must never change when attributes change, otherwise every downstream mart breaks at once.

3) Unity Catalog structure. A dedicated catalog (or at least schema) for conformed dimensions works well, for example a "conformed" catalog owned by a platform/data-governance group. Grant consumers SELECT only, and give the owning group MANAGE. Mart teams get full control in their own catalogs, so they can innovate without touching the shared layer. This is the cleanest answer to the governance versus agility trade-off: strict where it is shared, free where it is local.

4) Schema evolution without cascading failures. Version the contract, not the table. Additive changes (new nullable columns) are free. Breaking changes (rename, type change, dropped column) go out as a new view version (dim_customer_v2) while v1 is kept for a deprecation window. Marts should read through views, not the base table, so you can shim old shapes. Use lineage (system.access.table_lineage) to see exactly which consumers are on v1 before you retire it.

5) Contract testing. Yes, and it pays off the most of anything here. Put expectations on the producer side (not null and unique on the surrogate key, no overlapping validity windows, row count within a tolerance of the previous run), and have the consumer repo run a small test against the dimension's schema in CI. If the producer pipeline fails its expectations, it should not publish the new version, so marts keep reading the last good one.

Lesson learned: the hard part is rarely the technology, it is agreeing on who owns Customer. Settle the ownership and the change process first, and the Unity Catalog setup follows naturally.

[tamara48wells]

To implement a conformed dimension bus architecture in a Databricks Lakehouse, teams typically build master shared dimensions in a dedicated "Master Silver" layer—governed in a centralized Unity Catalog schema with strict row/column-level ACLs—while allowing downstream business domains read-only access to consume them into their Gold marts. To prevent cascading failures from schema evolution, you should enforce strict data contracts using automated tools like Great Expectations or Delta Live Tables expectations, and handle high-volume SCD Type 2 updates efficiently using Delta Lake's MERGE operations combined with change data feed (CDF).