Unity Catalog + Medallion Architecture
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-24-2024 07:55 PM
I am looking for a reference architecture or an example on how to organize unity catalog while adhering to the medallion architecture.
What are some common naming conventions and methods?
How to you isolate environments (dev/prod)?
I was thinking of something like this:
- One "catalog" per environment.
- One "schema" per set of tables with gold/silver/bronze versions
Example:
lakehouse_dev (catalog)
- bronze_widgets (schema)
-- table1_bronze, table2_bronze, etc
- silver_widgets (schema)
-- table1_silver, table2_silver, etc
- gold_widgets (schema)
-- table1_gold, table2_gold, etc
lakehouse_prod (catalog)
- bronze_widgets (schema)
-- table1_bronze, table2_bronze, etc
- silver_widgets (schema)
-- table1_silver, table2_silver, etc
- gold_widgets (schema)
-- table1_gold, table2_gold, etc
I'd love get get some other peoples thoughts on how they have implemented this in their organization. Thanks!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-25-2024 10:55 AM
Advancing Analytics have a good (although contrarian) take on the medallion architecture: Behind the Hype - The Medallion Architecture Doesn't Work - YouTube
I found that vid very helpful.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-27-2024 10:51 PM
Hi Shane,
Could you share what you ended up doing in your scenario. I have similar requirements and would like to understand how you implemented yours
Saurabh

