topic Serverless compute returns 403 on S3 buckets in the account-regional namespace in Get Started Discussions

Serverless compute returns 403 on S3 buckets in the account-regional namespace

alexgfowler_ld — Sun, 20 Sep 2026 09:10:34 GMT

SUMMARY

Serverless SQL warehouses and serverless notebook/jobs compute cannot read S3 buckets created in the AWS account-regional namespace (names ending -{accountId}-{region}-an). Classic compute reads the same buckets through the same IAM role without trouble. The failure is a 403 Forbidden with a null S3 request ID, raised by the bundled aws-sdk-java/1.12.681.

Measured across two days, same region, same role, same credential, same file, with the reads interleaved seconds apart:

Global namespace buckets: 15 of 15 reads succeeded

Account-regional namespace buckets: 0 of 14 reads succeeded

ENVIRONMENT

Cloud: AWS

Workspace and UC metastore: ap-southeast-1 (Singapore)

Affected compute: Serverless SQL warehouse (PRO, 2X-Small), serverless notebook/jobs compute

Unaffected compute: Classic job clusters, non-serverless PRO SQL warehouse

Client named in errors: Hadoop 3.5.0, aws-sdk-java/1.12.681

Storage access: Unity Catalog storage credential over a cross-account IAM role, external locations, managed and external tables

THE SYMPTOM

Any serverless read of an object in an account-regional bucket fails at the first

getFileStatus call:

[UNAUTHORIZED_ACCESS] Unauthorized access: s3://my-bucket-ACCOUNTID-ap-southeast-1-an/bronze/.../_delta_log: getFileStatus on s3://my-bucket-ACCOUNTID-ap-southeast-1-an/bronze/.../_delta_log: shaded.databricks.awssdk.com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden; request: HEAD https://my-bucket-ACCOUNTID-ap-southeast-1-an.s3-ap-southeast-1.amazonaws.com Hadoop 3.5.0, aws-sdk-java/1.12.681 ... credentials-provider: ...BasicSessionCredentials credential-header: AWS4-HMAC-SHA256 Credential=ASIA...../20260919/ap-southeast-1/s3/aws4_request signature-present: true (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: null; S3 Extended Request ID: null; Proxy: 192.168.200.20) SQLSTATE: 42501

Two details are worth noting immediately.

1. Request ID: null. A genuine IAM denial from S3 returns a real request ID. We have one in our records from a deliberately under-permissioned probe. A null request ID is the signature of this problem and the fastest way to tell the two apart.

2. signature-present: true, with a real ASIA session credential correctly scoped to the

region. Credentials were vended and the request was signed. This is not a

missing-credential problem.

FINDING 1: ACCOUNT-REGIONAL BUCKETS FAIL, GLOBAL-NAMESPACE BUCKETS SUCCEED

AWS lets you create a general-purpose bucket in either the shared global namespace or your account-regional namespace. The latter requires the name to end -{accountId}-{region}-an and is created with --bucket-namespace account-regional. It is a security feature: the name can only ever be claimed by your account, so it cannot be re-registered by a stranger after deletion.

Serverless cannot read them.

Evidence: Two buckets created 90 seconds apart in ap-southeast-1, holding a byte-identical file,

accessed through the same IAM role, the same Unity Catalog storage credential, and the

same external-location and volume mechanism. The only difference is the namespace.

Measured across two separate days, on cold warehouse sessions, with the two reads

interleaved seconds apart:

Global namespace: 15 of 15 successful reads

Account-regional namespace: 0 of 14 successful reads

Both credentials were tested against both buckets, in both directions, and the result never moved. The same holds for real Delta tables, not just probe files. A catalog whose managed location is a global-namespace bucket serves a 1,634-row Delta table to serverless without issue, while the identical data in an account-regional bucket returns 403 in the same session.

What this is not

We eliminated these by test, not by argument.

IAM permissions. Classic compute reads and writes the same bucket through the same role. The two bucket policies involved are structurally identical, same actions including s3:ListBucket, differing only in the bucket ARN.
The storage credential. Two different UC storage credentials, both over the same role, give identical results on both buckets. Neither credential is impaired: both succeed elsewhere.
Unity Catalog configuration. A brand-new external location, volume, schema and catalog reproduce it. A brand-new external location over a working bucket succeeds. Object age is not a factor.
Bucket configuration. Identical AES256 default encryption, no bucket policy, BucketOwnerEnforced, identical public access block on both.
Workspace identity. Reproduced on a second, independently created workspace with its own VPC.
Access method. Fails through external tables, managed tables and UC volumes alike.

FINDING 2: WHY ONE REGION IS UNAFFECTED, AND THE SDK ANGLE

ap-southeast-5 (Malaysia, GA August 2024) is unaffected even for account-regional buckets. Same account, same role, same credential. It simply works, and still does.

The bundled client named in every error is aws-sdk-java/1.12.681

Extracting com/amazonaws/partitions/endpoints.json from that exact artifact shows its partition metadata is dated 2024-03-15, five months before ap-southeast-5 became generally available, and the region is absent from it entirely.

That gives a coherent, testable hypothesis: regions known to the pinned SDK route through

a legacy client path that does not handle account-regional bucket addressing, and regions

unknown to it fall through to a different path that does. The AWS SDK for Java v1 final

release (1.12.797, December 2025) does contain ap-southeast-5, so if that is the

mechanism, refreshing the dependency would move ap-southeast-5 into the failing set rather

than fixing anything.

We cannot verify this from outside. We cannot inspect which client handles the request,

and no supported Spark configuration on serverless lets us influence the storage path. We

offer it as a starting point, not a conclusion. The measured facts are that one region

works, others fail, and the difference tracks that SDK release date.

CONFOUNDING FACTORS, PLEASE READ BEFORE REPRODUCING

These cost us several wrong conclusions.

1. A newly created S3 bucket is unreadable from serverless for up to about 2 hours

Regardless of namespace, and with the identical 403. We measured a brand-new bucket failing at 2 minutes, 27 minutes and 42 minutes old, then succeeding consistently from about 2 hours. Create two buckets, test them immediately, and both fail, leading to the confident and wrong conclusion that namespace is irrelevant. Three of our own experiments were voided this way. It is consistent with S3 virtual-hosted-style DNS propagation, given the failing requests use the legacy s3-REGION.amazonaws.com form.

Wait at least 2 hours after bucket creation before drawing any conclusion.

2. Control-plane credential validation passes regardless

Validating the storage credential returns READ, LIST and PATH_EXISTS all PASS on locations

that serverless compute cannot read. Validation exercises the role without the compute

session scoped-down policy. A green validation is not evidence.

3. Direct file queries swallow the real error

Querying a path directly returns a generic FAILED_TO_CREATE_PLAN_FOR_DIRECT_QUERY. Use an

external table, a managed table, or a UC volume. Otherwise the underlying S3 exception

never surfaces and you cannot tell this apart from anything else.

4. Single trials are not enough

We once observed a global-namespace bucket succeed twice and then fail 90 seconds later,

at the tail of the propagation window above. Run at least three trials per configuration

and treat any inconsistency as a result in itself.

5. S3 server access logging will not answer whether the request reached S3

We enabled it and planted a known-good control request. The control never appeared in the

delivered logs either, so the absence of the failing requests proves nothing. Server access

logging is best-effort by design. Use CloudTrail S3 data events if you need this question

answered.

6. Unity Catalog itself is fine

Listing catalogs, schemas, tables, external locations and storage credentials all work on

serverless, and lineage is captured normally. Only the reading of objects out of S3 fails.

Do not go hunting a Unity Catalog misconfiguration.

REPRODUCTION

Step 1. Create two buckets that differ only in namespace.

ACCOUNT_ID=your-aws-account-id REGION=ap-southeast-1 AN=repro-$ACCOUNT_ID-$REGION-an GL=repro-global-$ACCOUNT_ID aws s3api create-bucket --bucket $AN --region $REGION \ --create-bucket-configuration LocationConstraint=$REGION \ --bucket-namespace account-regional aws s3api create-bucket --bucket $GL --region $REGION \ --create-bucket-configuration LocationConstraint=$REGION

Any region that predates roughly March 2024 should show the behaviour.

Step 2. Put an identical file in each.

printf 'id,label\n1,test\n' > /tmp/c.csv aws s3 cp /tmp/c.csv s3://$AN/files/c.csv --region $REGION aws s3 cp /tmp/c.csv s3://$GL/files/c.csv --region $REGION

Step 3. Grant your existing Unity Catalog role read access to both buckets: s3:GetObject

on the object ARN, plus s3:ListBucket and s3:GetBucketLocation on the bucket ARN.

Step 4. In Unity Catalog, create one external location per bucket using the same storage

credential, and an EXTERNAL volume over each.

Step 5. Wait at least 2 hours. Then, on a serverless SQL warehouse or serverless notebook:

SELECT count(*) FROM read_files('/Volumes/CAT/SCH/global_vol/', format => 'csv'); SELECT count(*) FROM read_files('/Volumes/CAT/SCH/account_regional_vol/', format => 'csv');

The first returns 1. The second returns 403 with Request ID: null.

Control: point the same two queries at a non-serverless warehouse or a classic cluster. Both succeed. That is what localises the fault to serverless compute rather than to IAM, the bucket, or Unity Catalog.

WORKAROUNDS WHILE THIS IS UNPATCHED

1. Use a global-namespace bucket for anything serverless must read.

This is the only real fix available to customers today. Verified end to end: a catalog with

its managed location on a global-namespace bucket in the same region serves Delta tables to

both serverless SQL warehouses and serverless notebooks, with data loaded and written from

serverless too.

The cost is real and worth stating plainly. The account-regional namespace exists so that a

bucket name can never be claimed by another AWS account. Moving to the global namespace

gives that up. The namespace also cannot be changed in place, so this is a data migration,

not a rename.

2. Use classic compute for existing account-regional buckets.

Classic job clusters and non-serverless PRO SQL warehouses read them without trouble. The

trade-offs are minutes rather than seconds to start, a 10-minute minimum idle timeout on

non-serverless warehouses, and cluster management you may not want.

One trap worth knowing: a workflow task that omits its job-cluster setting silently falls

back to serverless and then fails at run time. Assert the setting in whatever you use to

define jobs.

3. Keep an eye on fine-grained access control.

Databricks documents a dedicated classic cluster as delegating row filters, column masks

and dynamic views to the workspace serverless compute. If that path applies, a table with a

row filter could fail on a classic cluster that otherwise works. We have no such filters and

have not tested this. Flagging it because classic compute is otherwise the safe ground.

A SEPARATE ISSUE WE HIT ALONG THE WAY: TRIAL ACCOUNTS AND SAME-REGION S3

Independent of the above, and worth recording because the two are easy to conflate. We lost time doing exactly that.

While our account was on a trial, serverless could not read any customer bucket in the workspace own region, global-namespace buckets included. Cross-region buckets were unaffected:

ap-southeast-5, account-regional, customer bucket A -> works ap-southeast-5, global, customer bucket B (new) -> works ap-southeast-1, global, customer bucket C -> 403 ap-southeast-1, account-regional, customer bucket D -> 403

Bucket C, a global-namespace bucket in the metastore own region, returned 403 during the trial. After the trial concluded, the same bucket, unchanged, reads reliably. Bucket D still fails, which is Finding 1.

Databricks documents serverless as reaching same-region S3 through a Databricks-owned private gateway, and different-region S3 out through NAT. A trial-level network restriction on the private gateway path would produce exactly the table above while leaving the NAT path alone. Two staff posts on this forum state that trial accounts carry network restrictions regardless of configured policy settings, and the serverless SQL warehouse requirements state the account must not be on a free trial.

This is inference, not measurement. We cannot put the account back on trial to test it. The before and after on bucket C is measured, and the trial end date is confirmed in writing. The mechanism is not.

Note the shape carefully. It is not that trial accounts cannot use customer S3 buckets. Cross-region customer buckets worked fine throughout. Stating it too broadly sent us chasing the wrong mechanism for days.

WHAT WOULD HELP FROM DATABRICKS

1. Confirm whether account-regional namespace buckets are supported by serverless compute. If not, this belongs in the documented limitations. The namespace is a first-class AWS S3 feature and nothing currently warns against it.

2. Identify the component returning the 403. The request is signed, the credential is valid, and the same address answers correctly when the request comes from other compute, but the response carries no S3 request ID.

3. Comment on the SDK hypothesis in Finding 2, which we cannot test from outside.

4. Confirm whether a trial account restricts same-region serverless S3 access. If it does, that belongs in the published trial limitations. We could find nothing stating it.

Re: Serverless compute returns 403 on S3 buckets in the account-regional namespace

Louis_Frolio — Fri, 02 Oct 2026 14:41:32 GMT

Greetings @alexgfowler_ld, I did some digging and here is what I found.

First, this is one of the most carefully built-out reports I've seen on this forum. Your A/B test already rules out the usual suspects: same role, credential, data, and region work from classic compute, and serverless fails only for the account-regional bucket. That's not an IAM or Unity Catalog setup problem. I can't see inside the serverless network path (nobody outside engineering can), but I can confirm what's confirmable and point you at the right door.

What AWS says

Account regional namespaces launched March 12, 2026, and the S3 user guide says these buckets support every S3 feature with no application changes required. So from the AWS side, nothing about the bucket should behave differently. This is something in the path between serverless compute and S3, not an S3 limitation.

Where the 403 is likely coming from

Your read on the null request ID is right. S3 stamps every response it generates, errors included, with x-amz-request-id and x-amz-id-2. Both are null and the exception names a proxy, so the simplest explanation is that the proxy in the serverless network path minted the 403 and the request never reached S3. Same-region S3 traffic from serverless rides a VPC gateway endpoint inside Databricks-managed VPCs (the firewall docs describe this), and cross-region traffic takes a different route, which is why ap-southeast-5 kept working during your trial.

On the SDK hypothesis

It holds together. The failing request went to the legacy dash-style hostname (bucket.s3-ap-southeast-1.amazonaws.com), which older partition data hands out for regions it knows, while an unknown region falls back to s3.REGION.amazonaws.com. So the two populations really do take different hostname shapes. I wouldn't call it root cause yet, though, and I'd hold off on the conclusion that S3 refuses these buckets on the legacy hostname: if S3 were saying no, you'd have a request ID. More likely a component in the serverless path matches on hostname and handles the two shapes differently. One detail to put in front of support: these bucket names now contain a region string and an account ID, so anything that parses names or hostnames to work out region or ownership has a fresh way to get it wrong. Also, the AWS SDK for Java 1.x reached end of support on December 31, 2025, so a client pinned to it was never going to learn about a bucket feature that shipped in March 2026.

Your other questions

Trial accounts: the serverless SQL warehouse requirements state plainly that the account must not be on a free trial. The mechanism isn't documented, so I can't confirm it's the same-region path specifically, but your before-and-after on bucket C is clean evidence and I agree it belongs in the published trial limitations.

Fine-grained access control: your caution is correct. On dedicated access mode compute, any query touching a table with row filters or column masks, or a dynamic view, is handed to the workspace's serverless compute for filtering, so it would fail even though the cluster itself can read the bucket. Standard access mode compute and pro SQL warehouses evaluate those controls themselves, so that's the safer ground in the meantime. Your note about tasks with no compute setting falling back to serverless is also correct.

Support status: the serverless limitations page says nothing about bucket namespaces either way. I don't know the answer, and the public docs don't establish it as unsupported.

Workarounds and next step

Your two workarounds are the right ones: classic compute for existing account-regional buckets, or a global-namespace bucket for anything serverless must read, after weighing the security trade-off you already described. Then please open a support case at help.databricks.com. Your post is most of the ticket already. Include the repro, the null request ID observation, the ap-southeast-5 contrast, the compute types and timestamps, and the exact hostnames in the failing versus working requests (that last one is most likely to shorten the investigation). Redact the account ID in bucket names. Only the serverless team can see what the proxy is doing with these requests, and if it turns out to be a gap, you've already written the documentation ticket.

References:

AWS, namespaces for general purpose buckets: https://docs.aws.amazon.com/AmazonS3/latest/userguide/gpbucketnamespaces.html
AWS, account regional namespaces announcement: https://aws.amazon.com/about-aws/whats-new/2026/03/amazon-s3-account-regional-namespaces
Databricks, serverless compute firewall configuration: https://docs.databricks.com/aws/en/security/network/serverless-network-security/serverless-firewall-config
Databricks, set up serverless SQL warehouses: https://docs.databricks.com/aws/en/compute/sql-warehouse/serverless
Databricks, serverless compute limitations: https://docs.databricks.com/aws/en/compute/serverless/limitations
Databricks, fine-grained access control on dedicated compute: https://docs.databricks.com/aws/en/compute/single-user-fgac
Databricks, connect to an AWS S3 external location: https://docs.databricks.com/aws/en/connect/unity-catalog/cloud-storage/s3
AWS, end of support for AWS SDK for Java 1.x: https://aws.amazon.com/blogs/developer/announcing-end-of-support-for-aws-sdk-for-java-v1-x-on-december-31-2025/

Regards, Louis.