Enterprise-level Unity Catalog governance

[APJESK]

Hi everyone

I'm currently designing an enterprise-level Unity Catalog governance model for Databricks and would like to learn from others who have implemented it in production.

I'm trying to understand what personas are actually required at each level, such as:

Enterprise / Account
1.Metastore
2.Catalog
3.Schema
4.Table
5.RBAC (role-based access control)
6.ABAC (attribute-based access control)
Some questions I have are:

1. What personas do you use at each level?
2. Who owns the catalog, schema, and tables?
3. Who manages RBAC permissions?
4. Who manages ABAC policies, governed tags, row filters, and column masks?
5. Do you have separate personas for Data Owners, Data Stewards, Security, and Platform teams?
6. How do you avoid too many groups and permission sprawl?
7. What has worked well in your enterprise implementation?
   If anyone has designed or implemented Unity Catalog governance in a large enterprise, I'd really appreciate it if you could share your architecture, persona model, or any best practices and lessons learned.

[balajij8]

You can design the Unity Catalog governance model at the enterprise level keeping the catalog topology simple and strictly enforcing group-based ownership. 

At the top level, you need Account Admins (2-3 people from the core cloud or platform infrastructure team) to manage identity integration and link meta stores. Meta store Admins should be used sparingly & dont rely on them for day-to-day operations. You can keep this centralized group extremely tight and delegate standard operations down to Workspace Admins or manage top-level objects like external locations and catalogs via a dedicated governance Service Principal using Terraform.

When it comes to ownership at the Catalog and Schema levels, it must be tied to cloud synced groups (like Azure AD). You can scope your catalogs by environment and domain (like sales_prod). The Catalog Owner should be a broader administrative group like a domain data engineering lead or platform DevOps team. They manage the top-level permissions but shouldn't be micromanaging individual tables. More details here

Schemas are the actual team boundaries. The Schema Owner (like a specific data engineering team) controls the objects within. By granting USE SCHEMA and CREATE TABLE exclusively to that team's group, you prevent random developers from cluttering shared namespaces.

For production tables, ownership should always fall to the Service Principal running the automated pipeline. Engineers should never own production tables. Data consumers like analysts, data scientists or BI tools can get SELECT access via functional groups.

To handle self-service discovery without exploding your permission models, grant BROWSE on the catalog to all users. It allows users to see table metadata and schema definitions without exposing the underlying data. You can scale the architecture by shifting from RBAC to ABAC for data security. Rather than creating dozens of localized groups to hide specific columns or rows, centralize the policy management.

• Security/Compliance Team writes the actual row filter and column mask functions at the enterprise level.

• Data Stewards are responsible only for applying and maintaining governed tags (like pii or gdpr_sensitive) on the assets within domains.

Enforce a strict group naming convention in the cloud active directory before syncing via SCIM (like <domain>_<role>_<scope> sales_analyst_prod). Dont grant privileges directly to individual users or emails. You can set up a monitoring job to periodically query the system.access.audit tables to catch and flag any manual grants that bypass the group architecture.

More details here

[binlogreader]

@APJESK -- @balajij8 's model is close to what we run in production, so I'll only add the mechanics we learned along the way.

Question 2, who owns catalog, schema, and tables. Catalogs and schemas are owned by the platform and data engineering group, created and managed only through Terraform, so ownership is a property of code, not of whoever clicked first. Production tables are owned by the service principal that runs the pipeline writing them. Deployment tooling stamps ownership on what it deploys, and transferring ownership away from a human requires elevated rights, so human-owned production objects eventually break CI deploys outright. Do the one-time transfer to the CI service principal early, not when it forces you.

Question 3, who manages RBAC. Exactly one Terraform stack owns each securable's grants, and every other team changes access by pull request to that stack. The rule exists because `databricks_grants` is authoritative for the object it points at. Two stacks declaring grants on the same catalog silently revert each other on every apply, and the symptom is grants disappearing at random. Humans get access only through IdP-synced groups, so the daily "who" is whoever manages group membership in the IdP, and the grants code changes only when a new group-to-privilege relationship appears.

Question 6, avoiding sprawl. The single-owner rule above is also the anti-sprawl mechanism. Every grant has one reviewable home, so a new group or privilege has to survive a pull request. Consumers sit in a small set of functional groups with SELECT plus BROWSE, never per-user grants. And manual UI grants get caught without a monitoring job, because anything added by hand shows up as drift in the owning stack's next plan, where it gets codified or reverted. The UI becomes break-glass only.

Questions 1 and 5, personas. At our scale, tens of pipelines and a handful of teams, the list collapses to three. The platform and data engineering group owning catalogs and schemas through code, domain teams owning what lives in their schemas, and consumers in functional groups. Separate steward and security personas earn their existence once governed tags and ABAC policies are actively maintained; we have not run ABAC or governed tags in production.

[APJESK]

I'm designing an enterprise Unity Catalog RBAC model in Databricks and would like feedback on whether this follows best practices.

My current design is:

• A single Service Principal is used by Terraform to provision all Unity Catalog objects.

• The Service Principal is a member of the Metastore Admin group.

• Using Terraform, the Service Principal:

  • Creates the Storage Credentials and External Locations.

  • Creates the Catalog, then transfers ownership to the appropriate Catalog Owner group.

  • Creates the Schema, then transfers ownership to the appropriate Schema Owner group.

• The Service Principal remains the automation identity, while business ownership is transferred to the respective owner groups.

My RBAC roles are:

• Metastore Admin

• Catalog Owner

• Schema Owner

• Schema Write

• Schema Read

Is this considered a good enterprise-scale design and aligned with Databricks Unity Catalog best practices?

Specifically:

1. Is it a best practice to use a single Terraform Service Principal that belongs to the Metastore Admin group for provisioning?

2. Should the Metastore Admin group continue to own Storage Credentials and External Locations, while Catalog and Schema ownership is transferred to the respective owner groups?

3. Are these RBAC roles sufficient for a large enterprise, or would you recommend adding or changing any roles?

[emma_s]

@APJESK, the existing replies from @balajij8 and @binlogreader cover the practical patterns well, in answer to your questions. But generally your design looks good with a few tweaks.

1. Single Terraform SP as Metastore Admin for provisioning - yes, this is a good approach.

2. Storage Credentials and External Locations owned by Metastore Admin group - again the right appraoch

3. Your RBAC roles (Metastore Admin, Catalog Owner, Schema Owner, Schema Write, Schema Read) - yes but also consider the following:

• Give browse to all users at the catalog level. This means your users will be able to see what data is there without actually accessing it, making it easy for them to request access to what they need.
• Consider a Schema Manage role (or grant MANAGE to Schema Owner groups). The MANAGE privilege delegates ownership-like abilities without transferring actual ownership, which is useful when multiple teams need to administer grants within a schema. 
• Production table ownership should go to the pipeline SP, not to human groups. Reserve direct MODIFY access to production tables for service principals only. This will only be the case though if your production datasets are read only. You may also want some kind of sandbox environment where analysts can write to some kind of scratch schema.

I hope this helps. 

Many Thanks,
Emma