Cross-region S3 reads fail with 400 Bad Request on Serverless

[harsh120]

Problem
We read Delta tables from an S3 bucket in af-south-1 (opt-in region) from a Databricks workspace in ap-south-1. Queries work on a classic cluster but fail on Serverless / SQL warehouse with the same data and the same IAM instance profile.

Same-region reads (bucket in ap-south-1) work on both classic and Serverless.

The error

getFileStatus on s3a://<BUCKET>/prod/<schema>/master-data/<table>/_delta_log:
com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request;
request: HEAD https://<BUCKET>.s3.af-south-1.amazonaws.com prod/<schema>/master-data/<table>/_delta_log
Hadoop 3.4.2, aws-sdk-java/1.12.681
(Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request)

400 Bad Request with no proper S3 error code (e.g. AccessDenied, InvalidArgument) — similar to this thread about eu-west-1 → af-south-1 cross-region reads.

Setup

• Workspace region: ap-south-1
• Bucket region: af-south-1 (account opt-in enabled)
• Catalog: hive_metastore (not Unity Catalog for these tables)
• Auth: Instance profile on classic cluster (arn:aws:iam::<account>:role/<role-name>)
• UC: No external location registered for the cross-region bucket
• Serverless network policy: Default — “Allow access to all destinations”

Reproduces by compute type

Compute DBR / runtime Cross-region (af-south-1) Same-region (ap-south-1)

Classic cluster	16.4 LTS	Works	Works
Serverless / SQL warehouse	Serverless	400	Works

So this looks specific to Serverless + cross-region opt-in region, not IAM or missing data (_delta_log exists in S3).

What we tried

1. Bucket policy on the cross-region bucket (same pattern as the working in-region bucket) — no change on Serverless
2. SET spark.hadoop.fs.s3a.bucket.<BUCKET>.endpoint in SQL editor — INVALID_PARAMETER_VALUE: not a supported SQL configuration parameter
3. CREATE STORAGE CREDENTIAL — [PARSE_SYNTAX_ERROR] (UC DDL not available in our SQL environment)
4. Checked Serverless network policy — already allow all destinations (so not a restricted egress policy)

On classic cluster, cross-region reads work without extra Spark S3A config.

What we ruled out

• Missing _delta_log / corrupt Delta files (verified in S3)
• IAM permissions (classic cluster reads same paths with same instance profile)
• Restrictive Serverless network policy (open / allow-all)
• Single-table issue (all tables on the cross-region bucket fail on Serverless)

Proof it’s the region pair

• Tables on in-region bucket (ap-south-1) → Serverless OK
• Tables on cross-region bucket (af-south-1) → Serverless 400
• Same tables on classic cluster (DBR 16.4 LTS) → OK

Question
The eu-west-1 → af-south-1 thread suggests SigV4 signing region mismatch and fs.s3a.bucket <BUCKET>.endpoint / endpoint.region as a classic-cluster workaround — but Serverless blocks spark.hadoop.fs.s3a.* configs.

Is cross-region S3 access to af-south-1 from Serverless in ap-south-1 supported for hive_metastore external Delta tables? If yes, what’s the supported configuration (UC external location, service credential, something else)?

If anyone has hit this pattern — classic works, Serverless 400 on opt-in region bucket — we’d appreciate any workaround or escalation path.