topic Re: Data Dictionary and Unity Catalog: Best Practices in a Medallion Architecture in Data Governance

Data Dictionary and Unity Catalog: Best Practices in a Medallion Architecture

faruk — Fri, 26 Jun 2026 06:35:24 GMT

Hi everyone,

I'm currently working on implementing a data dictionary, and I'm struggling to understand what the best approach is when using a Medallion architecture in Databricks.

My company is migrating its data to Databricks so that users can consume data directly from the platform instead of querying our production databases. At the same time, we want to build a data dictionary that both technical and business users can use to quickly understand and discover available data.

Initially, I thought it would be as simple as maintaining an Excel file. However, I quickly realized that this approach isn't scalable. We have several databases, hundreds (if not thousands) of schemas, thousands of tables, and an even larger number of columns. On top of that, with a Medallion architecture (Bronze, Silver, Gold), documenting every version of every column across all layers would become impossible to maintain over time.

Our current idea is to leverage Databricks as the single source of truth by storing column comments directly in Unity Catalog and propagating them across the different layers whenever possible. However, I've noticed that metadata propagation is not always reliable. In addition, tags don't seem to propagate along with comments, which makes governance more difficult.

I'm also unsure how to document information such as expected values or valid value ranges. Should these be captured as SQL CHECK constraints (when applicable), stored as tags, included in the column comments, or managed somewhere else?

More generally, I'm having trouble finding best practices for implementing a data dictionary in a Medallion architecture. How do you organize metadata across the Bronze, Silver, and Gold layers? Do you maintain a single business definition that is shared across layers, or do you document each layer independently?

Our long-term goal is to build a complete data catalog with a business glossary. We'd also like to expose this metadata to AI tools in the future, so having well-structured and well-governed metadata is becoming increasingly important.

I'd be interested to hear how others have approached this problem or what best practices you would recommend.

Re: Data Dictionary and Unity Catalog: Best Practices in a Medallion Architecture

balajij8 — Fri, 26 Jun 2026 07:31:39 GMT

You can follow below to create a sustainable layered governance framework across the Medallion architecture separating technical metadata from business semantics.

Layered Strategy

Bronze Layer (Technical & Operational) - Keep the documentation minimal and automated. You can focus strictly on upstream lineage, source systems, ingestion frequencies and raw formats (Ingested via DLT from SAP ERP). Keep the business definitions at a high level or try to avoid as schemas can change upstream and as users generally don't consume raw data.
Silver Layer (Conformed & Classified) - Business context begins here. You can ensure standardized schemas (naming, types, time zones etc) and use column comments to explicitly document data cleaning or transformation rules applied. You can introduce Governed Tags here for data classification (PII, sensitive, public etc) to drive downstream security policies.
Gold Layer (Consumption Ready Semantics) - Its the main & primary documentation target for business users and BI tools. Ensure that every column has a rich business definition aligned with the organization’s master business glossary. Maintain comprehensive compliance tag coverage for data ownership and discovery.

Unity Catalog Execution

Unity Catalog provides various features to support the governance.

Column Comments - You can focus on adding calculation logic, edge cases and provide concrete examples such as order_status_code: The operational lifecycle state of the order. Valid values: PENDING, SHIPPED, DELIVERED, CANCELLED.
Governed Tags - Use tags as structured & queryable metadata. You can establish an upfront taxonomy (e.g., Data_Classification: PII, Business_Domain: Finance) and then leverage these tags directly in security policies based on roles.
Constraints - You can use CHECK constraints strictly to actively enforce data quality (as CHECK (age >= 0)). Document the rationale behind the constraint inside the column comment.

AI Consumption

You can follow below to ensure Unity Catalog's metadata is useful for AI cases

Enforce Structured Formats - Use deterministic patterns in column comments (such as always ending with Valid values: [X, Y, Z]) allowing LLM parsers to accurately extract categorical boundaries.
Define Relationships - Ensure primary and foreign key relationships are explicitly defined in the schemas. UC uses these to help AI engines write accurate multi-table joins.
Publish Freshness Metrics - You can attach metadata regarding data freshness, SLA targets and update frequencies to your table properties to allow AI models know if they are pulling stale data.

Re: Data Dictionary and Unity Catalog: Best Practices in a Medallion Architecture

JeninaAngelin7 — Fri, 26 Jun 2026 07:41:33 GMT

Hey @faruk , great question. But the answer depends on your data's current state. Are you migrating from DBFS mount paths to UC tables, or do you already have Hive metastore tables that you're migrating to UC?

Check out dbxmetagen — a Databricks solution accelerator that helps generate metadata at scale using LLMs. Could help seed your metadata table faster.

External vs Managed tables — document which is which. It affects your governance model significantly. This matters for permissions, disaster recovery, and preventing accidental data loss.

Managed = UC controls the lifecycle (drop table = data gone).
External = you control storage (drop table = data stays).

On column mapping mode — if your schema is strict and you don't expect columns to be renamed or dropped, you don't need delta.columnMapping.mode = 'name'. Keep it as default. It's only worth enabling if you anticipate schema evolution (renames, drops).

A manual metadata table works well here. Build your own Delta table as a data dictionary — columns like catalog, schema, table_name, column_name, business_description, valid_values, owner, etc. This becomes your controlled source of truth.

Good to go?

Re: Data Dictionary and Unity Catalog: Best Practices in a Medallion Architecture

emma_s — Fri, 26 Jun 2026 14:29:02 GMT

Hi,

This is a challenging area to get right, especially with so many tables and schemas. Unity Catalog is definitely a good bet for storing this metadata. There are a few approaches that you could take to the data definitions part, and it depends on what you need from them.

At the moment, I would recommend using the comments on fields and table definitions to store possible values and definitions. This could be done with a fixed JSON structure to make it readable, but this depends on whether you need queryable metadata or not. If it's just to be read by AI and humans, you probably don't need it to be so structured.

Comments Docs - https://docs.databricks.com/aws/en/comments/

You can also use tags, but it could become cumbersome for all the different columns to have tags for the definitions etc which is why I’d probably stick to the comments. I may use tags for specific things like PII, etc. You can also mark Primary keys for tables where needed.

Tag Docs - https://docs.databricks.com/aws/en/database-objects/tags

In terms of propagation, comments don’t generally propagate; usually, I’d expect some sort of transformation between layers, so comments propagating doesn’t really work. I’d focus your efforts on the layers that will be consumed by humans and AI, which are usually your gold layers. These would then represent your final business layer. As others have said, AI can help with this, but be careful, good metadata should be potentially things the AI will not necessarily be able to infer, and therefore, using AI to infer it may leave gaps.

The final things to keep an eye on are the Genie Ontology and the UC Glossary; these have just been announced. Genie ontology learns context from your notebooks and dashboard, and could, over time, make some of your metadata less necessary.

https://www.databricks.com/product/unity-catalog/business-semantics

My general recommendation would be to pick one schema and map through that first, and this will help to inform your approach and think about how you want to tackle it going forward.

I hope this helps.

Thanks,

Emma

Re: Data Dictionary and Unity Catalog: Best Practices in a Medallion Architecture

faruk — Sat, 27 Jun 2026 10:33:27 GMT

Hi , thank you for your advice very appreciate that

Re: Data Dictionary and Unity Catalog: Best Practices in a Medallion Architecture

faruk — Sat, 27 Jun 2026 10:34:16 GMT

@JeninaAngelin7 Thank you for your answer, very instructive.

Re: Data Dictionary and Unity Catalog: Best Practices in a Medallion Architecture

faruk — Sat, 27 Jun 2026 11:20:20 GMT

Hi @emma_s,

First of all, thank you for your detailed answer.

As you mentioned, tagging every column — and in my case every field — could quickly become overwhelming.

I’ve decided to implement tags only for specific use cases, as you suggested. For allowed values and definitions, I see that many people use the comment section, so I will likely follow the same approach. I’m also considering using metric views to enrich the business context, and potentially, once we reach a higher level of maturity, migrating to a dedicated data catalog solution like DataHub. That would make it easier to extract and manage all this metadata consistently.

Regarding AI usage on top of our data, I’m still unsure whether focusing only on the gold layer is the right approach for us. In some cases, we also need to access raw or less aggregated data directly, depending on the use case.

Thanks a lot,

Kr,

Faruk

Re: Data Dictionary and Unity Catalog: Best Practices in a Medallion Architecture

JeninaAngelin7 — Sun, 28 Jun 2026 15:01:15 GMT

Yeah Sure, Welcome!

Re: Data Dictionary and Unity Catalog: Best Practices in a Medallion Architecture

Rjdudley — Tue, 30 Jun 2026 20:45:13 GMT

We are implementing Atlan to provide a business glossary to users and agents. We're ingesting multiple sources, primarily Databricks and SQL Server based. For data with a published definition we copy those into table and column comments. We also use Databricks' comment generation tooling with manual review. We do this in Databricks so Genie also has the ontology. For SQL Server we use Atlan's similar definition creation capabilities. Atlan also provides us a more complete lineage since sources and sinks of data often exist outside of Databricks.

Definitions don't often carry through from bronze to silver to gold because the data meaning is changing along the way, especially in gold metrics. We define the data in all layers.