Hi everyone,

I'm currently working on implementing a data dictionary, and I'm struggling to understand what the best approach is when using a Medallion architecture in Databricks.

My company is migrating its data to Databricks so that users can consume data directly from the platform instead of querying our production databases. At the same time, we want to build a data dictionary that both technical and business users can use to quickly understand and discover available data.

Initially, I thought it would be as simple as maintaining an Excel file. However, I quickly realized that this approach isn't scalable. We have several databases, hundreds (if not thousands) of schemas, thousands of tables, and an even larger number of columns. On top of that, with a Medallion architecture (Bronze, Silver, Gold), documenting every version of every column across all layers would become impossible to maintain over time.

Our current idea is to leverage Databricks as the single source of truth by storing column comments directly in Unity Catalog and propagating them across the different layers whenever possible. However, I've noticed that metadata propagation is not always reliable. In addition, tags don't seem to propagate along with comments, which makes governance more difficult.

I'm also unsure how to document information such as expected values or valid value ranges. Should these be captured as SQL CHECK constraints (when applicable), stored as tags, included in the column comments, or managed somewhere else?

More generally, I'm having trouble finding best practices for implementing a data dictionary in a Medallion architecture. How do you organize metadata across the Bronze, Silver, and Gold layers? Do you maintain a single business definition that is shared across layers, or do you document each layer independently?

Our long-term goal is to build a complete data catalog with a business glossary. We'd also like to expose this metadata to AI tools in the future, so having well-structured and well-governed metadata is becoming increasingly important.

I'd be interested to hear how others have approached this problem or what best practices you would recommend.