topic Re: Unity Catalog + Medallion Architecture in Data Engineering

Unity Catalog + Medallion Architecture

shane_t — Thu, 25 Jan 2024 03:55:23 GMT

I am looking for a reference architecture or an example on how to organize unity catalog while adhering to the medallion architecture.

What are some common naming conventions and methods?

How to you isolate environments (dev/prod)?

I was thinking of something like this:

- One "catalog" per environment.

- One "schema" per set of tables with gold/silver/bronze versions

Example:

lakehouse_dev (catalog)

- bronze_widgets (schema)

-- table1_bronze, table2_bronze, etc

- silver_widgets (schema)

-- table1_silver, table2_silver, etc

- gold_widgets (schema)

-- table1_gold, table2_gold, etc

lakehouse_prod (catalog)

- bronze_widgets (schema)

-- table1_bronze, table2_bronze, etc

- silver_widgets (schema)

-- table1_silver, table2_silver, etc

- gold_widgets (schema)

-- table1_gold, table2_gold, etc

I'd love get get some other peoples thoughts on how they have implemented this in their organization. Thanks!

Re: Unity Catalog + Medallion Architecture

JN_Bristol — Thu, 25 Jan 2024 18:55:01 GMT

Advancing Analytics have a good (although contrarian) take on the medallion architecture: Behind the Hype - The Medallion Architecture Doesn't Work - YouTube

I found that vid very helpful.

Re: Unity Catalog + Medallion Architecture

ssharma — Thu, 28 Nov 2024 06:51:35 GMT

Hi Shane,

Could you share what you ended up doing in your scenario. I have similar requirements and would like to understand how you implemented yours

Saurabh