Louis_Frolio
Databricks Employee
Databricks Employee

yesterday

Greetings @alexgfowler_ld, I did some digging and here is what I found.

First, this is one of the most carefully built-out reports I've seen on this forum. Your A/B test already rules out the usual suspects: same role, credential, data, and region work from classic compute, and serverless fails only for the account-regional bucket. That's not an IAM or Unity Catalog setup problem. I can't see inside the serverless network path (nobody outside engineering can), but I can confirm what's confirmable and point you at the right door.

What AWS says

Account regional namespaces launched March 12, 2026, and the S3 user guide says these buckets support every S3 feature with no application changes required. So from the AWS side, nothing about the bucket should behave differently. This is something in the path between serverless compute and S3, not an S3 limitation.

Where the 403 is likely coming from

Your read on the null request ID is right. S3 stamps every response it generates, errors included, with x-amz-request-id and x-amz-id-2. Both are null and the exception names a proxy, so the simplest explanation is that the proxy in the serverless network path minted the 403 and the request never reached S3. Same-region S3 traffic from serverless rides a VPC gateway endpoint inside Databricks-managed VPCs (the firewall docs describe this), and cross-region traffic takes a different route, which is why ap-southeast-5 kept working during your trial.

On the SDK hypothesis

It holds together. The failing request went to the legacy dash-style hostname (bucket.s3-ap-southeast-1.amazonaws.com), which older partition data hands out for regions it knows, while an unknown region falls back to s3.REGION.amazonaws.com. So the two populations really do take different hostname shapes. I wouldn't call it root cause yet, though, and I'd hold off on the conclusion that S3 refuses these buckets on the legacy hostname: if S3 were saying no, you'd have a request ID. More likely a component in the serverless path matches on hostname and handles the two shapes differently. One detail to put in front of support: these bucket names now contain a region string and an account ID, so anything that parses names or hostnames to work out region or ownership has a fresh way to get it wrong. Also, the AWS SDK for Java 1.x reached end of support on December 31, 2025, so a client pinned to it was never going to learn about a bucket feature that shipped in March 2026.

Your other questions

Trial accounts: the serverless SQL warehouse requirements state plainly that the account must not be on a free trial. The mechanism isn't documented, so I can't confirm it's the same-region path specifically, but your before-and-after on bucket C is clean evidence and I agree it belongs in the published trial limitations.

Fine-grained access control: your caution is correct. On dedicated access mode compute, any query touching a table with row filters or column masks, or a dynamic view, is handed to the workspace's serverless compute for filtering, so it would fail even though the cluster itself can read the bucket. Standard access mode compute and pro SQL warehouses evaluate those controls themselves, so that's the safer ground in the meantime. Your note about tasks with no compute setting falling back to serverless is also correct.

Support status: the serverless limitations page says nothing about bucket namespaces either way. I don't know the answer, and the public docs don't establish it as unsupported.

Workarounds and next step

Your two workarounds are the right ones: classic compute for existing account-regional buckets, or a global-namespace bucket for anything serverless must read, after weighing the security trade-off you already described. Then please open a support case at help.databricks.com. Your post is most of the ticket already. Include the repro, the null request ID observation, the ap-southeast-5 contrast, the compute types and timestamps, and the exact hostnames in the failing versus working requests (that last one is most likely to shorten the investigation). Redact the account ID in bucket names. Only the serverless team can see what the proxy is doing with these requests, and if it turns out to be a gap, you've already written the documentation ticket.

References:

Regards, Louis.

0 Kudos