Managing Service Principal Permissions at Scale: External Locations vs. Managed Volumes

[Khasim_1]

Hi community,

I’m currently architecting a multi-workspace environment under Unity Catalog, and I’m looking for the most sustainable way to manage Service Principal access to raw data landing zones.

We are debating between using External Locations (scoped to specific S3/ADLS paths) versus Unity Catalog Managed Volumes for our initial ingestion layer.

1. For those managing thousands of daily ingestion files, have you found Managed Volumes to be "production-ready" for high-volume service principal writes, or do you still prefer the granular control of External Locations?
2. How are you handling Cross-Workspace access for these service principals without creating a permission-management nightmare?
3. Are you using Terraform or Databricks Asset Bundles (DABs) to automate the GRANT cycles, and have you hit any API limitations when scaling this across hundreds of service principals?

I want to avoid building a "governance tax" that slows down our development velocity. Any advice on the "Cleanest" enterprise-grade approach?

[ivanvyd]

1. I'd benchmark your upload method, file sizes and peak concurrency before making a throughput claim. Daily file count alone isn’t enough.

I'd use managed volumes when access goes through Databricks, and external volumes for existing S3/ADLS landing paths written by external producers. The volume guide explains that distinction. Producers using their own cloud credentials still need appropriate IAM policies or ACLs; UC grants do not govern those direct requests.

For external volumes, register landing-zone subdirectories and grant access to those volumes rather than broad READ FILES / WRITE FILES permissions on the external location. This follows the Unity Catalog recommendations.

2. It is a good practice to grant permissions to account groups by workload and environment, then add service principals with matching access requirements. This avoids repeating object grants for every principal. Use account groups, not workspace-local groups, which cannot receive UC privileges.

For an existing account group and volume:

GRANT USE CATALOG ON CATALOG prod
TO `prod-vendor-ingestors`;

GRANT USE SCHEMA ON SCHEMA prod.landing
TO `prod-vendor-ingestors`;

GRANT READ VOLUME, WRITE VOLUME
ON VOLUME prod.landing.vendor_drop
TO `prod-vendor-ingestors`;

Replace the example names and run this using an identity authorized to manage the grants. The volume privileges reference confirms these requirements for both managed and external volumes. No external-location grant is needed for file access to an existing volume. WRITE VOLUME includes updates and deletes, so this is not append-only access.

For workspaces sharing a metastore, reuse the volume and its grants. Assign each principal to the workspaces it needs, and use catalog bindings to restrict where the data is accessible. A read-only binding blocks writes even when the principal has write privileges.

3. I prefer keepingp shared UC resources and grants in Terraform, with application jobs and pipelines in bundles. Bundles also support volumes and their grants, so this is about choosing which deployment manages each object, not a product limitation.

The databricks_grants reference warns that the resource manages all grants on its target object and overwrites changes made elsewhere. Keep one authoritative definition per object rather than competing definitions across workspace states.

For custom REST automation, I’d limit concurrency and follow the UC API guidance: retry HTTP 429 responses with exponential backoff and jitter. I wouldn’t promise a safe request rate without testing it. Group-based grants also reduce the work: onboarding another principal with the same access requirements becomes a membership change, not another set of grants across every volume.

[Islam_hoti · Accepted solution]

Hi,

On question one, I would reframe the choice rather than answer it directly. Managed volumes sit in Databricks controlled storage, which means an external producer cannot write into them through Unity Catalog. If your landing zone is fed by systems outside Databricks, and it sounds like it is, external locations are not the more granular option, they are the only workable one. Put an external location over the path and create an external volume on top if you want volume semantics for downstream code. Managed volumes are a good fit when Databricks itself is the writer, which is a different layer of your architecture.

On question two, the premise may be causing more work than it needs to. Unity Catalog privileges are granted at the metastore level, not per workspace, so a grant is already cross workspace by default. What you actually want is the opposite control: restricting exposure. That is workspace binding. The default isolation mode is open to every workspace attached to the metastore, and you can set catalogs and external locations to isolated and bind them to specific workspaces. So the pattern is grant once centrally, then limit which workspaces can see the object.

The thing that removes most of the governance tax, though, is not granting to service principals at all. Grant to groups, and manage membership instead. Hundreds of service principals becomes a handful of groups such as ingest writers, raw readers, and so on. This also makes your API scaling concern in question three largely disappear, because you are managing a small number of grant objects and a larger number of group memberships, and membership changes are much cheaper than permission changes.

On tooling, the split that has worked for people is Terraform for governance objects that live at the account and metastore level, meaning storage credentials, external locations, catalogs, groups and grants, and Asset Bundles for the workloads that run on top. Bundles can carry grants for the Unity Catalog resources they define, but trying to make them the source of truth for metastore wide permissions tends to fight the tool. Keeping the governance state in one Terraform state file also gives you the audit trail that will be asked for eventually.