Hello,

I tried your suggestion by setting up the peering connection between the 2 VPC but issue remains the same.

The error message

The bucket is in this region: .... please use this region to retry the request

makes me think that the root cause is not at network level but at S3 Select Spark connector level which does not use correct regional s3 enpoint.

The connector does not seem to have such property : https://docs.databricks.com/external-data/amazon-s3-select.html doc

Then I tried to set following properties at Spark level as usually suggested in such situation without any effect :

spark.conf.set("fs.s3a.endpoint","s3.eu-west-1.amazonaws.com")

spark.conf.set("fs.s3n.endpoint","s3.eu-west-1.amazonaws.com")

spark.conf.set("fs.s3.endpoint","s3.eu-west-1.amazonaws.com")

It seems that the S3 Select connector does not forward this endpoint setting to the underlying AWS S3 SDK