topic Re: Unity Catalog for medallion architecture in Data Engineering

Unity Catalog for medallion architecture

mbravonxp — Sat, 08 Feb 2025 11:53:04 GMT

Hello community.

I need help to define the most suitable approach for Unity Catalog. I have the following storage architecture in Azure Data Lake Storage.

I have data from different clients
I work with 3 different environments for each client: dev, pre, pro
I need to implement a medallion architecture, bronze, silver and gold for each environment

I need to read, write and work with the Data from Databricks. The Azure data storage gets updated with new data on a daily basis.

What would be the best approach considering catalogs, schemas, external volumes, tables and so on?

Thanks in advance.

Re: Unity Catalog for medallion architecture

Alberto_Umana — Sat, 08 Feb 2025 22:25:42 GMT

Hello @mbravonxp,

Create a separate catalog for each client to logically isolate their data. This helps in managing permissions and organizing data efficiently.

Within each catalog, create schemas for each environment (dev, pre, pro). This will help in managing the data lifecycle and access control for different stages of development

Implement the medallion architecture by creating separate schemas or tables within each environment for the bronze, silver, and gold layers. This will help in organizing the data processing pipeline and maintaining data quality.

Bronze: Raw data ingestion.
Silver: Cleaned and enriched data.
Gold: Aggregated and business-level data.

Use Databricks jobs or workflows to automate the data processing and updating of tables in the bronze, silver, and gold layers

Best Practices:

Avoid giving direct storage-level access to users for Unity Catalog managed tables or volumes to maintain data security and governance.
Co-locate your Databricks workspace, metastore, and storage in the same Azure region for optimal performance

Re: Unity Catalog for medallion architecture

filipniziol — Tue, 11 Feb 2025 07:42:12 GMT

Hi @mbravonxp ,

in this case the best approach is to have a single catalog per client per environment (so 9 catalogs in total).
In every catalog you will create bronze, silver and gold schema.
Additionally every catalog will have a separate storage and also, you may consider to have a separate workspace for each client for each environment.

Check my answer to the similar topic on the forum:
https://community.databricks.com/t5/community-platform-discussions/unity-catalog-implementation/td-p/84082

Re: Unity Catalog for medallion architecture

mbravonxp — Wed, 12 Feb 2025 08:25:56 GMT

Hi both,

Thanks very much for the useful replies. Definitely I will go for your suggestions.

Best.