topic Re: Azure Databricks Unity Catalogue Limits in Data Governance

Azure Databricks Unity Catalogue Limits

sqlshep — Fri, 29 Sep 2023 20:52:28 GMT

In Azure, what are the limits of Unity Catalogue? One UC per region per subscription, or is it one UC per region per tenant?

Re: Azure Databricks Unity Catalogue Limits

btafur — Sat, 30 Sep 2023 02:37:05 GMT

One metastore per region per tenant is an enforced soft limit that works for nearly all organisations. You have an additional level of isolation that can be achieved by creating Catalogs inside the Metastore. The recommendation is to use those Catalogs to achieve the isolation you need. You can divide your Catalogs per environment (DEV, Staging, PROD), Business Unit, a mix of both or however you find more useful to your needs.

Each catalog can have a separate LOCATION to store their data in different buckets/containers (S3/ADLS) if required to separate the data in the storage. Also, you can isolate which Catalogs can be accessed by which workspaces, giving you full flexibility.

https://docs.databricks.com/en/data-governance/unity-catalog/best-practices.html

Re: Azure Databricks Unity Catalogue Limits

sqlshep — Sat, 30 Sep 2023 16:30:07 GMT

thank you for the response, you mentioned "soft limit" so is this something that can be changed?

Re: Azure Databricks Unity Catalogue Limits

SiddeshSamarth — Fri, 26 Jan 2024 15:00:29 GMT

Does it mean we will have DEV, TEST, PROD catalog in a single Unity catalog?

Re: Azure Databricks Unity Catalogue Limits

satycse06 — Thu, 13 Aug 2026 12:15:42 GMT

In most of the implementation I have seen the different metastore for prod and dev i.e one metastore for dev and test and another separate metastore for prod but in that case databricks account is also different for those environments.

For the cloud the tenant will be same but the subscription will be different.

The real logic is how you assign your databricks workspace to the metastore. At a time one workspace can be part of only one metastore but one metstaore can be part of multiple workspace.

And it's also depends on how the data isolation you want and at which level? If there is requirement to have the multiple metastore as per the different domain data or due to some other compliance rule then you can have it.