Unity Catalog external location access error after metastore attachment

[Khasim_1]

Hi everyone,

I am running into a permission issue when trying to query data from an external S3 storage location managed through Unity Catalog.

Here are the symptoms:

• The Setup: Storage credentials and an external location were successfully created, and the metastore is properly linked to the workspace.
• The Problem: When running a query or trying to list files in the external location, the job fails with a permission denied error, even though the assigned IAM role appears correct on the cloud side.
• What I'm looking for: Has anyone faced a similar authorization gap between AWS IAM and Unity Catalog storage credentials? What troubleshooting steps or logs should I check next?

Any guidance from the community would be greatly appreciated!

[data_pulse]

We hit something similar. In our case the job ran as a service principal that was missing the required UC privilege, even though the IAM role was fine. For path-based access that’s READ FILES on the external location, for tables it’s SELECT, USE CATALOG/USE SCHEMA and for volumes it’s READ VOLUME. Granting the right one fixed it.

Few common causes and triage steps:

• UC privileges: Confirm the job’s identity and its grants, and that the path falls under the external location

   SHOW GRANTS ON EXTERNAL LOCATION my_location;
   DESCRIBE EXTERNAL LOCATION my_location;​

if privilege is missing, then

GRANT READ FILES ON EXTERNAL LOCATION my_location TO `user_or_sp`​

• Compute: Make sure the job runs on UC-enabled compute (standard/ dedicated access mode / serverless).
• Validation: Run Validate Configuration on the storage credential and Test connection on the external location in Catalog Explorer, then test directly. Also try:

DESCRIBE STORAGE CREDENTIAL my_credential;
LIST 's3://my-bucket/path/';​

• IAM and AWS: Check the trust policy and external ID (including self-assuming role), S3 permissions and KMS key policy if encrypted.
• Workspace binding: Check the external location’s bindings, especially if the metastore association changed recently via Catalog Explorer -> Connect -> External Location or with Databricks CLI

databricks external-locations get my_location

databricks workspace-bindings get-bindings external-location my_location​

[pradeep_singh]

A correct IAM role is necessary but not sufficient. Access to an external location goes through two separate checks, and the wording of the error tells you which one failed. Here's how to find it quickly.

Step 0 – Capture the exact error. Run a direct path read from the same compute and identity that fails:

LIST 's3://<bucket>/<prefix>/';

Then match the message:

Error text containsFailing layerGo to

PERMISSION_DENIED: User does not have READ FILES on External Location ... (or USE CATALOG / USE SCHEMA / SELECT)	Unity Catalog privileges	Step 1
"not accessible in the current workspace" / location greyed out in Catalog Explorer	Workspace binding	Step 2
"AWS IAM role does not have READ permissions on url ...", "403 Forbidden from cloud storage provider", "Failed on assuming role"	AWS (trust, S3, KMS, bucket policy)	Step 4
Errors about the Unity API token, or access mode	Compute	Step 3

Step 1 – Unity Catalog privileges (the most common cause). A storage credential does not grant users anything. The calling principal needs privileges on the external location, and for tables, on the catalog, schema and table too. For a job, that means the job's run-as identity.

SHOW GRANTS ON EXTERNAL LOCATION `my_location`;

GRANT READ FILES  ON EXTERNAL LOCATION `my_location` TO `data_engineers`;

GRANT WRITE FILES ON EXTERNAL LOCATION `my_location` TO `data_engineers`;  -- writes need READ FILES too

-- for external tables / volumes on that path:

GRANT USE CATALOG ON CATALOG my_cat TO `data_engineers`;

GRANT USE SCHEMA  ON SCHEMA  my_cat.my_sch TO `data_engineers`;

GRANT SELECT ON TABLE my_cat.my_sch.my_tbl TO `data_engineers`;   -- or READ VOLUME on a volume

Notes:

• A principal with WRITE FILES but no READ FILES still gets PERMISSION_DENIED on writes.
• Databricks recommends granting READ VOLUME on a volume rather than READ FILES on raw paths for most users.

Step 2 – Workspace bindings. If the location (or its catalog) is set to isolated, it only works from bound workspaces.

databricks external-locations get my_location            # check isolation_mode

databricks workspace-bindings get-bindings external-location my_location

Either add your workspace ID, or tick "All workspaces have access" on the location's Workspaces tab.

Step 3 – Compute. Use a UC-capable access mode:

• Standard (formerly Shared) or Dedicated (formerly Single user) clusters, a SQL warehouse, or serverless.
• "No isolation shared" and legacy clusters can't use UC external locations.

Step 4 – AWS side. Start with Catalog > External data > Credentials > your credential > Validate configuration, then Test connection on the external location. The results show which step fails: self-assume, read, list, write or delete. Then check, in this order:

1. Trust policy: it must trust the Databricks UC master role ARN shown in the docs or on the credential page, and the role itself (self-assuming), with sts:ExternalId equal to the external ID shown on the credential page. Databricks blocks non-self-assuming roles. If you recreated the role, re-copy the external ID.
2. Permissions policy: s3:ListBucket and s3:GetBucketLocation on arn:aws:s3:::<bucket>, and object actions (GetObject, plus PutObject/DeleteObject for writes) on arn:aws:s3:::<bucket>/<prefix>/*. A common mistake is scoping ListBucket to the /* object ARN.
3. KMS (SSE-KMS buckets): the role needs kms:Decrypt/GenerateDataKey/Encrypt, and the KMS key policy must name the role.
4. Bucket policy explicit Deny: look for conditions such as aws:SourceVpce, aws:SourceVpc, aws:PrincipalOrgID or encryption headers.
• An explicit Deny beats every Allow.
• UC metadata and validation calls can come from the Databricks control plane rather than your workspace VPC endpoint. A "deny unless my VPCE" policy can therefore break UC even when your clusters reach S3.
• If the policy requires encryption headers, set the encryption algorithm on the external location.
5. SCPs and the bucket ACL: an Organizations SCP can deny S3 or KMS actions. Also, the docs require that the bucket has no S3 ACL attached; use Object Ownership = Bucket owner enforced.

Logs to check:

• AWS CloudTrail: filter on the role ARN, looking at AssumeRole events and S3/KMS AccessDenied events. The errorMessage usually names the denying policy type (identity, resource, SCP or VPC endpoint). S3 data events must be enabled to see object-level denies.
• Databricks: system.access.audit with service_name = 'unityCatalog' shows which principal and which action UC evaluated:

SELECT event_time, user_identity.email, action_name, request_params, response.status_code, response.error_message

FROM system.access.audit

WHERE service_name = 'unityCatalog'

  AND event_date >= current_date() - 1

  AND response.status_code <> 200

ORDER BY event_time DESC;

If Validate configuration passes, the bucket policy has no Deny, and you still get a 403, open a support ticket with the request ID from the error. That case needs a server-side look at the assume-role attempt.

[saurabh18cs]

Hi @Khasim_1 

Are you doing direct file access or via unity catalog tables/volumes? if you are reading directly then you need to grant READ FILES to the identity used by your pipeline/notebook to this external location.

If you are accessing via uc table/volumes then you dont need this READ FILES.. then you have to make sure proper permissions are granted to catalog/schema/table for the same identity either user or sp running this notebook/job. (e.g. USE CATALOG, USE SCHEMA, SELECT or READ on table/volume

this should solve your issue , if not let me know. Thanks

[Louis_Frolio]

Hello @Khasim_1, I took a look at both internal and external documentation and here is what I found.

First, credit to @data_pulse, @pradeep_singh and @saurabh18cs, who've already laid out the standard triage. Capture the exact error. Confirm the job's run-as identity (user or service principal) has the right UC grants: READ FILES on the external location for direct path reads, or USE CATALOG, USE SCHEMA and SELECT (or READ VOLUME) for tables and volumes. Check the workspace binding if the location is isolated, and make sure the compute is UC-capable (Standard, Dedicated, SQL warehouse or serverless). Pradeep's error-text-to-layer table is the fastest way to find out which of those you're hitting.

I'll add the pieces that may add challengs right after a metastore gets attached.

The role on your cluster probably isn't the one being checked. Before UC, S3 access usually rode on the cluster's instance profile. Once an external location covers a path, UC enforces access to that path on UC-enabled compute and ignores cluster-level credentials and Spark configs. The role that matters is the one inside the storage credential, and the Databricks control plane assumes it, not your cluster. So the cluster role can look perfect in AWS and tell you nothing.

On that credential role, check three things:

• The trust policy has the real external ID. The documented flow creates the role with a placeholder (0000), then has you copy the external ID from the "credential created" dialog and go back to update the trust policy. Skip that last step and the credential still shows as created, but every assume-role call fails.
• The role trusts itself. Since January 2025 Databricks blocks storage credentials whose role isn't self-assuming, so an older role that used to work can fail today.
• Run Validate Configuration on the credential and Test Connection on the external location in Catalog Explorer. They exercise the same assume-role, list, read and write calls UC makes at query time, and the failing step points at the policy to fix (trust, S3 bucket and prefix, or the KMS key policy if the bucket uses SSE-KMS).

Grants don't transfer either. Attaching a metastore carries nothing over from instance profiles or legacy table ACLs, and UC grants only go to account-level users, groups and service principals. If your workspace predates UC and the grant went to a workspace-local group, it won't apply. SHOW GRANTS ON EXTERNAL LOCATION my_location shows who actually has what; compare it with the identity on the failed run.

Logs settle which side is at fault quickly. Query system.access.audit with service_name = 'unityCatalog' for the denied request, and check CloudTrail for the role. No AssumeRole call at all means UC blocked it before reaching AWS; an AssumeRole or S3/KMS AccessDenied means the problem is in AWS.

If you're mid-migration and need existing jobs running while you sort out grants, the external location owner can turn on fallback mode. UC checks the caller's grants first and, if those fail, falls back to the cluster's instance profile or Spark config credentials. Treat it as a temporary bridge and switch it off once the grants are in place.

References:

• Create a storage credential for S3 (trust policy, external ID, self-assuming): https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/storage-credentials
• Create an S3 storage credential and external location (Validate and Test Connection): https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3/s3-external-location-manua...
• Manage external locations (grants, workspace bindings): https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/manage-external-locations
• Unity Catalog privileges reference: https://docs.databricks.com/aws/en/data-governance/unity-catalog/access-control/privileges-reference
• Best practices for DBFS and Unity Catalog (cluster configs ignored for UC paths): https://docs.databricks.com/aws/dbfs/unity-catalog
• Fallback mode on external locations: https://docs.databricks.com/aws/en/query-federation/hms-federation-internal
• Compute access mode limitations for Unity Catalog: https://docs.databricks.com/aws/compute/access-mode-limitations
• Audit log system table: https://docs.databricks.com/en/admin/system-tables/audit-logs.html

Regards, Louis.

[Fastlearning272]

This type of issue can sometimes occur when AWS IAM permissions are correctly configured, but Unity Catalog is unable to authorize access through the storage credential or external location. I would recommend checking a few things before changing the IAM role.

Here are some troubleshooting steps worth exploring:

1. Verify Storage Credential Permissions: Make sure the storage credential is accessible to the user or service principal executing the query. Check whether the required READ FILES privilege has been granted.

2. Check External Location Grants: Verify that the user has the necessary permissions on the external location. For example, READ FILES is required to read or list files, while WRITE FILES is needed for write operations.

3. Validate IAM Trust Relationship: Check the AWS IAM role's trust policy and confirm that it allows the correct Databricks IAM principal to assume the role. Also verify the external ID if one is required.

4. Review S3 Bucket Policies: Even when the IAM role has the correct permissions, an S3 bucket policy, explicit deny statement, or restrictive VPC endpoint policy can block access. Check permissions for s3:ListBucket and s3:GetObject on the relevant bucket and objects.

5. Inspect CloudTrail Logs: AWS CloudTrail can help identify whether the request is being denied by AWS. Look for AccessDenied events and check which principal is making the request.

6. Check Unity Catalog Configuration: Confirm that the external location points to the correct S3 path and uses the intended storage credential. Also check for overlapping external locations or conflicting storage paths.

7. Test with a Simple SQL Command: Try listing files using:
   LIST 's3://your-bucket/your-path/';

This can help narrow down whether the issue is related to basic storage access or a higher-level table permission.

One important point: AWS IAM authorization and Unity Catalog authorization are two separate permission layers. Both must allow the requested operation.

I would start by checking the Unity Catalog grants and AWS CloudTrail logs. These two checks can often help identify where the authorization failure is happening. 💩

Hope this helps someone facing a similar issue! 😊💼