Just for the record, this still does not work. I spent some more time trying to debug this with Genie as well, and we landed on the conclusion that we can't do anything else on the Premium plan at least. If I had the Enterprise Plan, I could have tried adding private endpoints in the NCC, and maybe that would have worked? I don't know.

But the bottom line is that custom S3 buckets can't be accessed as external locations using the serverless compute on the premium plan. I was using external Delta tables on S3, and it was not working with serverless compute, but using UC-managed Delta tables (also saved on S3) works fine on serverless compute as well, and there is no difference in any bucket or IAM policy between the two, so probably Databricks is doing something behind the scenes.

I am not sure if this is intentional or some issue, but it should be documented. I reached out to Databricks support, but they can't help me without a support contract with them and asked me to ask for help in the community. So this is it then.