https://community.databricks.com/t5/data-engineering/quot-amazons3exception-the-bucket-is-in-this-region-quot-error/m-p/28909#M20674 <P></P> <P>I have read access to an S3 bucket in an AWS account that is not mine. For more than a year I've had a job successfully reading from that bucket using dbutils.fs.mount(...) and sqlContext.read.json(...). Recently the job started failing with the exception: "com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: us-east-1. Please use this region to retry the request." on the sqlContext.read.json() command. My Databricks Cloud platform is in us-west-2, and the bucket may have been moved, but as far as I understand from this question this shouldn't be a problem: <A href="https://forums.databricks.com/questions/416/does-my-s3-data-need-to-be-in-the-same-aws-region.html" target="test_blank">https://forums.databricks.com/questions/416/does-my-s3-data-need-to-be-in-the-same-aws-region.html</A> . I have no problem accessing the bucket with boto3 (with no need to specify region). I've also tried setting the 'spark.hadoop.fs.s3a.endpoint' of the cluster to 's3.us-east-1.amazonaws.com' . Surprisingly this resulted in the same error on the mount() command saying that the bucket is in the 'us-west-2' region.</P> <P>I'm a bit confused as to the possible causes of this error and would be happy for some pointers,</P> <P>Thanks!</P> <P></P> Sun, 18 Feb 2018 11:57:10 GMT DanielAnderson 2018-02-18T11:57:10Z https://community.databricks.com/t5/data-engineering/quot-amazons3exception-the-bucket-is-in-this-region-quot-error/m-p/28909#M20674 <P></P> <P>I have read access to an S3 bucket in an AWS account that is not mine. For more than a year I've had a job successfully reading from that bucket using dbutils.fs.mount(...) and sqlContext.read.json(...). Recently the job started failing with the exception: "com.amazonaws.services.s3.model.AmazonS3Exception: The bucket is in this region: us-east-1. Please use this region to retry the request." on the sqlContext.read.json() command. My Databricks Cloud platform is in us-west-2, and the bucket may have been moved, but as far as I understand from this question this shouldn't be a problem: <A href="https://forums.databricks.com/questions/416/does-my-s3-data-need-to-be-in-the-same-aws-region.html" target="test_blank">https://forums.databricks.com/questions/416/does-my-s3-data-need-to-be-in-the-same-aws-region.html</A> . I have no problem accessing the bucket with boto3 (with no need to specify region). I've also tried setting the 'spark.hadoop.fs.s3a.endpoint' of the cluster to 's3.us-east-1.amazonaws.com' . Surprisingly this resulted in the same error on the mount() command saying that the bucket is in the 'us-west-2' region.</P> <P>I'm a bit confused as to the possible causes of this error and would be happy for some pointers,</P> <P>Thanks!</P> <P></P> Sun, 18 Feb 2018 11:57:10 GMT https://community.databricks.com/t5/data-engineering/quot-amazons3exception-the-bucket-is-in-this-region-quot-error/m-p/28909#M20674 DanielAnderson 2018-02-18T11:57:10Z https://community.databricks.com/t5/data-engineering/quot-amazons3exception-the-bucket-is-in-this-region-quot-error/m-p/28910#M20675 <P></P> <P>@andersource</P> <P> Looks like the bucket is in <PRE><CODE>us-east-1</CODE></PRE> but you've configured your <PRE><CODE>AmazonS3</CODE></PRE> Cloud platform with <PRE><CODE>us-west-2</CODE></PRE>. Can you try switching configuring the client to use <PRE><CODE>us-east-1</CODE></PRE> ?</P> <P>I hope it will work for you. Thank you</P> <P></P> Sat, 08 Jun 2019 06:59:23 GMT https://community.databricks.com/t5/data-engineering/quot-amazons3exception-the-bucket-is-in-this-region-quot-error/m-p/28910#M20675 Chandan 2019-06-08T06:59:23Z