- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-04-2024 06:40 AM
Hi @prad18,
I have experience working with Databricks on Azure, and here are some considerations regarding your Unity Catalog implementation:
Important Consideration:
The most important fact on Azure is that you can only have a single metastore per tenant per region. This means that when planning your Unity Catalog implementation, you need to design your architecture to work within this limitation. Let me address your questions, starting from the last one and working upwards.
3. Azure Subscription and Metastore Architecture:
Given the limitation of a single metastore per tenant per region, the architectural decisions around Unity Catalog need to reflect this constraint.
Single Metastore Implementation (Answer to 3.b):
Since you can only have one metastore per tenant per region, you will set up a single metastore. This single metastore will contain multiple catalogs. You will create separate catalogs within the metastore for each environment (e.g., `dev_catalog`, `staging_catalog`, `prod_catalog`).
2. Storage Account Allocation:
Separate Azure Storage Accounts:
- Unity Catalog Metastore vs. Actual Data:
- Metastore Storage Account: Use a dedicated storage account for Unity Catalog metadata. This storage is used to manage the metastore’s metadata and does not directly interact with your data processing tasks.
- Data Storage Account: Use separate storage accounts for your actual data. You may create a separate storage per catalog and then separate blob container for landing, bronze, silver and gold.
1. Workspace Separation Strategy:
Keep your workspaces separated. Have a separate workspace for production, a separate workspace for staging and a separate workspace for development. Staging may mirror production so it can contain production data. You do not want to have people doing the development and accidentally run their code targeting production data. Mixing development and staging might cause extra risks. In short best practice is to have separate workspaces per environment.
Summary:
- Single metastore with multiple catalogs. At least 1 catalog per environment.
- Separate storage accounts for metadata and data
- At least 1 workspace per environment
Last but not least: your organization may have multiple projects sharing the same unity metastore.
For example finance department has their data and sales department has their data.
In that case you want to create catalogs per environment per project, so in the example `finance_dev_catalog`, `finance_staging_catalog`, `finance_prod_catalog`, `sales_dev_catalog`, `sales_staging_catalog`, `sales_prod_catalog`.
Proper naming will make your life easier in future.