Your debugging is really thorough and you've already done the hard work of isolating this. The 400 with an empty body (no proper S3 error code like InvalidArgument) on an opt-in region is almost always one thing: SigV4 signing region mismatch. af-south-1 strictly enforces that the request is signed with af-south-1 as the signing region — if the SDK signs it with eu-west-1 or falls back to a global endpoint, S3 rejects it with exactly this signature.

Your observation that UC vending still returns credentials successfully but those credentials then fail is the giveaway. Something on the Databricks side recently changed how those STS credentials are being scoped or which STS endpoint is being used to generate them — regional STS tokens can carry implicit region restrictions that bite you on cross-region opt-in buckets.

Worth trying as a workaround — add these Spark configs at the cluster level (or session level to test first):

fs.s3a.bucket.<YOUR-BUCKET-NAME>.endpoint s3.af-south-1.amazonaws.com
fs.s3a.bucket.<YOUR-BUCKET-NAME>.endpoint.region af-south-1
This forces the S3A client to sign requests for that specific bucket using the correct region regardless of where the metastore sits. Bucket-scoped configs won't affect your other jobs.

That said — given this started on a specific date with no changes on your end and affects UC credential vending internally, you should also open a Databricks Support ticket and reference the date (2026-05-18 ~23:46 UTC). Your audit trail evidence is clean and this has all the hallmarks of a platform-side change to their STS/credential vending layer.

Let us know if the Spark config workaround helps in the meantime.