Coffee77
Honored Contributor III

It is a suggestion based on the chosen architecture in the main Databricks application I am currently working. Let me explain in detail:

- One workspace per environment, all of them governed by Unity Catalog with two metastores.

- Each workspace contains three different catalogs, to cover "bronze", "silver" and "gold" layers. All of them using delta tables.

- Discussing with our main cloud architect (MVP Microsoft), we came to the conclusion that placing each catalog in a dedicated Azure Datalake Storage Account (ADLS Gen2) would be much better for performance than having only one ADLS Gen2 with separated containers per environment or similar. Latter case, thoughput would be shared by all layers in medallion architecture. So, with dedicated layers, performance is much better in I/O operations.

- so, each layer has a related catalog and each catalog is physically placed ina different Azure Storage account.

I hope this helps.


Lifelong Solution Architect Learner | Coffee & Data