Getting 'Unauthorized Access' when Serverless compute is trying to read s3 bucket data

[primopragma]

Getting 'Unauthorized Access' when Serverless compute is trying to read s3 bucket data, same works fine with normal compute storage and external location, credentials, IAM policy and serverless egress rules all seems to be correct because normal compute is working. I am on a premium plan and have free credits for now, do we need enterprise plan for this? or is it something else?

This is the error I am getting when trying to list the s3 bucket items in serverless compute:

ExecutionError: [UNAUTHORIZED_ACCESS] Unauthorized access: s3://<raw-data-bucket>/<partition-path>: getFileStatus on s3://<raw-data-bucket>/<partition-path>: 
shaded.databricks.awssdk.com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; 
request: HEAD https://<raw-data-bucket>.s3-us-west-2.amazonaws.com/<partition-path> {} 
Hadoop 3.4.2, aws-sdk-java/1.12.681 Linux/6.1.174-217.345.amzn2023.x86_64 OpenJDK_64-Bit_Server_VM/17.0.18+8-LTS java/17.0.18 scala/2.13.16 kotlin/1.9.10 vendor/Azul_Systems,_Inc. cfg/retry-mode/legacy 
shaded.databricks.awssdk.com.amazonaws.services.s3.model.GetObjectMetadataRequest; 
Request ID: null, Extended Request ID: null, Cloud Provider: AWS, Instance ID: unknown 
credentials-provider: shaded.databricks.awssdk.com.amazonaws.auth.BasicSessionCredentials 
credential-header: AWS4-HMAC-SHA256 Credential=<redacted-access-key-id>/<date>/us-west-2/s3/aws4_request 
signature-present: true 
(Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: null; S3 Extended Request ID: null; Proxy: <redacted>), 
S3 Extended Request ID: null: 403 Forbidden

SQLSTATE: 42501

Caused by: com.databricks.sql.io.CloudAccessDeniedException: getFileStatus / listStatus 403 on same path as above.

[Trace ID: <redacted>]

[anagilla]

No, you don't need the Enterprise plan for this. Serverless reads Unity Catalog external locations fine on Premium; Enterprise only comes into play for optional serverless egress control features.

Your trace shows the request is signed and S3 itself returns the 403, so AWS is denying it, and the usual reason it denies serverless but not classic is the network path. Classic compute runs inside your VPC; serverless reaches S3 from Databricks-managed VPCs through a VPC endpoint. Any policy condition written for your VPC won't match serverless traffic.

Three things to check:

1. Bucket policy network conditions. Look for aws:SourceVpc, aws:sourceVpce, or source-IP conditions in the bucket policy. If you want to keep the bucket locked down, add a condition using aws:VpceOrgPaths with the Databricks organization path, per the serverless firewall doc.

2. KMS key policy. If the bucket uses SSE-KMS, the key policy must allow your storage credential's IAM role (kms:Decrypt, kms:GenerateDataKey*), per the S3 KMS doc.

3. Confirm classic is really using the external location. If your classic cluster has an instance profile attached, it may be that role doing the reading. Test on classic with no instance profile so you know the UC storage credential role itself has the S3 permissions.

[primopragma]

The trace I originally shared was a HEAD request (GetObjectMetadataRequest), which only returns headers, so the actual denial reason was hidden and it just showed a bare "Forbidden". Re-running the same access as a List operation surfaces the real error message in the response body.

Running this in a serverless notebook:

dbutils.fs.ls("s3://<raw-data-bucket>")

gives:

ExecutionError: [UNAUTHORIZED_ACCESS] Unauthorized access: s3://<raw-data-bucket>: getFileStatus on s3://<raw-data-bucket>: shaded.databricks.awssdk.com.amazonaws.services.s3.model.AmazonS3Exception: Access to storage destination is denied because of serverless network policy; request: GET https://<raw-data-bucket>.s3-us-west-2.amazonaws.com / ... ListObjectsV2Request; Request ID: null, Extended Request ID: null ... credentials-provider: BasicSessionCredentials ... signature-present: true (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: null; Proxy: <redacted>)

[Trace ID: <redacted — can share via DM/ticket>]

So the denial is explicitly "because of serverless network policy" — it's coming from the egress proxy, not from AWS IAM. A few supporting facts:

- The workspace's only network policy is set to "Allow access to all destinations" and hasn't changed recently
- system.access.outbound_network is empty
- UC vends session credentials fine (signed request in the trace), and Browse in Catalog Explorer lists the bucket
- From the same serverless session, dbutils.fs.ls on the Databricks-auto-created external location works — only this manually created one is denied. Both buckets are in us-west-2, same region as the metastore

This looks like the egress proxy's storage allowlist being out of sync with the UC external locations for my workspace. Happy to share the trace ID and workspace ID privately so you can check which allowlist snapshot the proxy evaluated.

[anagilla]

Nice debugging, that error string settles it: the denial is coming from the serverless egress layer, not AWS, and the auto-created vs manual split is the tell. Serverless auto-allows a bucket only when the external location is bound to the workspace or open to all workspaces, and binding changes can take up to 24 hours to take effect.

1. Open Catalog Explorer, go to the manual external location, Workspaces tab. If it's in isolated mode and this workspace isn't listed, bind it (or set it to all workspaces).

2. To unblock right away, add the bucket explicitly under Allowed Storage Destinations on the network policy in the Account Console. That's the documented way to expedite and propagates in about 10 minutes. While you're there, confirm which policy this workspace is actually attached to; the "allow all" one you checked may not be it.

3. If the binding looks right and it still fails after the propagation window, open a support ticket with the workspace ID and trace ID you offered. Support can check whether the bucket is registered on the egress side for your workspace, and the explicit allowlist entry keeps you running in the meantime.

Also worth re-running the system.access.outbound_network query later; system tables can lag, and the DROP rows with the rejection reason are the strongest evidence to attach to a ticket.

[primopragma]

Thanks — checked all three:

1. Workspaces tab on the external location shows "All workspaces have access", so it's not in isolated mode — it should be auto-allowlisted.

2. Can't use the explicit allowlist workaround — we're on the Premium tier, so managing storage destinations on the network policy isn't available to us.

3. Re-ran the system.access.outbound_network query — still empty, no DROP rows for this bucket yet. Will check again later in case the logs are lagging.

So the full state is: binding is all-workspaces, bucket and metastore are both us-west-2, IAM/trust/credential all validate (UC vends session credentials, Browse works, classic compute works), the only network policy is allow-all, and the auto-created external location works from the same serverless session while this manual one gets "denied because of serverless network policy".

One detail I should have mentioned earlier: both this workspace and the external location were created today. Could this simply be propagation? i.e., does the serverless egress allowlist for a brand-new workspace + newly created external location take up to 24 hours to converge — and would that also explain why the auto-created external location works (registered during workspace provisioning) while the manual one doesn't yet?

If that's expected behavior, I'll retest in fresh serverless sessions over the next day and report back. If it still fails after 24 hours, I'll open a support ticket with the workspace ID and trace ID.

[anagilla]

Your read is right, and fair point on the allowlist: editing network policies is part of serverless egress control, which is an Enterprise capability, so on Premium the propagation window is the path.

Your timing hypothesis matches how this works internally. Creating or updating an external location propagates quickly, but the workspace-level implicit registration that feeds the serverless storage allowlist converges on a slower cycle, documented at up to 24 hours. I'd say give it a try in sometime again.