Databricks Community

lbourgeois · 01-03-2023

Hello,I have a cluster running in us-east-1 region.I hava a Spark job loading data in a DataFrame using s3select format on a bucket in eu-west-1 region.Access and Secret keys are encoded in URI s3a://$AccessKey:$SecretKey@bucket/path/to/dirJob fails ...

lbourgeois · 01-26-2023

Hello,I tried your suggestion by setting up the peering connection between the 2 VPC but issue remains the same.The error message The bucket is in this region: .... please use this region to retry the requestmakes me think that the root cause is not ...

lbourgeois · 01-23-2023

Hi @Hubert Dudek and @47kappal ,Sorry for the delay. As suggested I'm trying to setup a gateway endpoint for s3 following https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.htmlI am a bit confused by A gateway endpoint is available...

lbourgeois · 01-04-2023

Sure, I reproduced the issue on a notebook. Here is the code snippet to create a Dataset with s3select and csv formats :val s3selectDS = spark.read.format("s3select").schema(mySchema) .load("s3://"+accessKey+":"+secretKey+"@lbourgeois-rd/s3selectdbr...

lbourgeois · 01-04-2023

Thanks @Hubert Dudek for having a look.I don't use unity catalog, I actually use 3.Encode keys in URI option for S3 Auth as described in https://docs.databricks.com/external-data/amazon-s3-select.html#s3-authenticationStrange thing is that if I chan...

Databricks Community

User Stats

User Activity

com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: *** when using S3 Select

Re: com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: *** when using S3 Select

Re: com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: *** when using S3 Select

Re: com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: *** when using S3 Select

Re: com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: *** when using S3 Select