anuj_lathi
Databricks Employee
Databricks Employee

Short answer: I'd build the golden record in a dedicated conformed layer that sits between Silver and the marts, rather than before Silver or inside any one mart. Treat that layer as a governed data product with a stable, versioned contract, and let the marts consume it read-only.

1. Where the golden record gets built

  • Silver stays source-aligned. It's cleansed, typed, deduplicated per source, and keyed on business keys. Keep cross-source survivorship and matching logic out of it.
  • Conformed dimensions are a separate step after Silver. Customer, Product, and Geography each get their own pipeline that does the entity resolution, survivorship rules, surrogate key assignment, and SCD2 history. It reads from multiple Silver sources and writes one table per dimension.
  • Domain marts only consume. They join facts to the conformed dimensions and never re-derive or fork them. If a mart needs an extra attribute, it goes into the conformed dimension through a change request, not into a local copy.

Doing this "before Silver" blurs ingestion cleansing with business-rule consolidation, and every Silver table then inherits cross-domain dependencies. Making each mart build its own version defeats the purpose of the bus architecture.

Also plan for late-arriving dimension members. Facts will show up before the dimension row exists, so insert inferred or placeholder members with stable surrogate keys and backfill them later. Dimension upserts are MERGE operations. Add the known constraints to the match condition, keep files compacted, and use Low Shuffle Merge to keep them cheap as dimensions grow [1].

2. Unity Catalog layout

There's no single right answer, but this is the pattern I use most:

prod_silver.<source_or_domain>.*        -- source-aligned
prod_conformed.core.*                   -- dim_customer, dim_product, dim_geography
prod_conformed.published.*              -- versioned views = the consumer contract
prod_gold_sales.marts.*                 -- domain marts (facts + mart-specific dims)
prod_gold_finance.marts.*
  • Separate catalog (or at minimum a separate schema) for conformed dimensions. This gives you a clean ownership boundary, so the conformed pipeline's service principal is the only principal with MODIFY on it.
  • Mart pipelines get USE CATALOG, USE SCHEMA, and SELECT only on the conformed objects. That makes the "no local forks" rule something you enforce rather than just ask for.
  • Separate dev/test/prod by catalog, so conformed-dimension changes can be validated against real mart pipelines before promotion.
  • Use Unity Catalog managed tables for the dimensions, with predictive optimization and liquid clustering rather than hand-tuned partitioning and maintenance jobs [1]. Dimensions are usually small to medium and benefit from the defaults.
  • Document the contract in Unity Catalog. Use table and column comments, tags (for example domain, certified, pii), and column masks or row filters where needed. Data stewards then see what's certified without reading pipeline code.

3. Change management without breaking consumers

Separate the physical table from the contract. Marts should read from the published views, not from core tables. That gives you room to refactor physical tables without touching consumers.

Classify changes:

Change Handling
Add a nullable column Non-breaking. Use Delta schema evolution on the conformed pipeline, then expose the column in the view when ready.
Rename, drop, or retype a column; change grain or key semantics Breaking. Publish a new version (dim_customer_v2), run both in parallel, announce a deprecation window, then retire v1.
Business-rule change (survivorship, matching) Treat as breaking if it changes surrogate keys or historical attributes. Otherwise treat as non-breaking but announce it.

Practical guardrails:

  • Keep surrogate keys stable. Never regenerate them on a rebuild, because every fact table depends on them.
  • Don't let upstream source schema drift flow straight through. Silver can evolve freely, but the conformed pipeline should select an explicit column list, so a new source column doesn't silently appear in a shared dimension.
  • Add Delta constraints (NOT NULL, CHECK) and pipeline expectations or data-quality tests on keys, uniqueness of current rows, and non-overlapping SCD2 date ranges. Fail the conformed pipeline rather than publish bad data to every mart.
  • If you need to redeploy a table in the same location, use CREATE OR REPLACE TABLE rather than drop and recreate [1].
  • Don't manually touch the underlying data files. Everything goes through Delta transactions [1].
  • Run CI for the conformed project that executes representative mart queries against the new version in a dev or test catalog before you promote.

4. Balancing single version of truth and agility

  • Centralize the small, high-value set. Only the truly shared entities, with a small number of attributes everyone agrees on, belong in the conformed layer.
  • Let marts own everything else. Mart-specific attributes, junk dimensions, and local hierarchies live in the mart schema and join to the conformed dimension on the surrogate key.
  • Run it like a product. Give it a named owner, a change-request process, a published release cadence, and the versioned-view contract. That's what lets mart teams move fast without negotiating every change.

If you're starting from scratch, I'd begin with one dimension (usually Customer or Geography), implement the full pattern of conformed pipeline, locked-down grants, published view, and CI test, and then repeat it for the rest.

References

[1] Best practices: Delta Lake | Databricks on AWS — https://docs.databricks.com/aws/en/delta/best-practices

Anuj Lathi
Solutions Engineer @ Databricks