Unity catalogues - What would you do

[Rezakorehi]

If you were creating Unity Catalogs again, what would you do differently based on your past experience?

[nayan_wylde]

That’s a great question. Unity Catalog have changed the entire data governance framework and I would do the following things to make UC projects successful based on my past experience of implementing  Unity Catalog.

1. Design the metastore hierarchy early on

• Initially, we created catalogs and schemas ad-hoc as teams onboarded.
• I’d establish a clear naming convention and ownership model before rollout — for example:
• Catalogs aligned with business domains (e.g., sales, finance, marketing)
• Schemas mapped to data zones or data products.
  This prevents rework and ensures consistent data lineage and access policies.

2. Implement access control as code from day one

• Early setups often involved manual grants using the Databricks UI or SQL commands.
• I’d now automate permission management using Terraform or Databricks APIs, SDK, DAB etc. treating Unity Catalog permissions as version-controlled code.
• This improves auditability and simplifies onboarding of new datasets.

3. Adopt a data ownership and stewardship model upfront

• Initially, central data teams managed all permissions, which quickly became a bottleneck.
• I’d define data owners and stewards per catalog or schema right from the start and delegate privileges accordingly through groups.Like make Business Owners manage the group permissions

4. Integrate Unity Catalog with external identity and policy systems early

• Integration with Azure AD SCIM for group-based access.

5. Plan for cross-workspace and cross-region access

• Unity Catalog enables sharing across workspaces, but initial setups often ignored multi-region or DR needs.
• I’d now design catalogs with data residency and cross-region replication in mind.

6. Involve stakeholders early

• Instead of just IT-led design, I’d include data producers, consumers, and compliance teams during catalog structuring to ensure business relevance and smooth adoption.

[Hubert-Dudek]

Hi, just always keep it on seperate storage container 🙂

[szymon_dybczak]

From my experiance:

- don't create separate catalogs for every project. Try to think about your desing before implementation

- try to come up with consistent naming convention to avoid cognitive overhead

- principle of least privilege - grant users and groups the minimum permissions necessary for their tasks, leveraging Unity Catalog’s hierarchical privilege inheritance.

[Hubert-Dudek]

@szymon_dybczak, I recently heard about a use case where the company is automatically creating a new catalog for every employee... but of course I agree with you. 

[nayan_wylde]

@Hubert-Dudek Is this a recommended way by databricks? Creating catalogs for each employees will only Complexity explosion: Hard to manage permissions, auditing, and lifecycle.
Performance overhead: Catalog metadata operations scale poorly with excessive catalogs. I would love to see how organisations tackle those challenges.

[szymon_dybczak]

Haha, that’s for sure an unconventional approach, but maybe they had a good reason for doing that 😃. In one of the projects I’ve participated in, we did a similar thing, but only for developers. They wanted to have an isolated development environment. 

[Hubert-Dudek]

@nayan_wylde no don't do that hehe. It was example of extreme approach. Usually use catalog to separate environment + in enterprises to separate divisions like customer tower, marketing tower, finance tower etc