Accessing the regions that are disabled by default in AWS from Databricks. In AWS we have 4 regions that are disabled by default. You must first enabl...

Prabakar — Fri, 22 Oct 2021 22:30:57 GMT

Accessing the regions that are disabled by default in AWS from Databricks.

In AWS we have 4 regions that are disabled by default. You must first enable it before you can create and manage resources. The following Regions are disabled by default:

Africa (Cape Town)
Asia Pacific (Hong Kong)
Europe (Milan)
Middle East (Bahrain)

In Databricks, when we try to access/mount the bucket in these regions it will fail. The reason is that the AWS SDK we bundle in Databricks doesn't have the by-default-disabled region information. So auto-region detection will not work for these regions.

In such a special case, to access the buckets in the disabled regions, we have to provide an S3 endpoint along with the STS endpoint to mount the bucket.

%scala
dbutils.fs.mount("s3a://<Bucket-in-Milan>/", "/mnt/milan/",
  extraConfigs = Map(
    "fs.s3a.credentialsType" -> "AssumeRole",
    "fs.s3a.stsAssumeRole.arn" -> "arn:aws:iam::000000000000:role/MyRole",
    "fs.s3a.acl.default" -> "BucketOwnerFullControl",
    "fs.s3a.endpoint" -> "s3.eu-south-1.amazonaws.com",
    "fs.s3a.stsAssumeRole.stsEndpoint" -> "sts.eu-south-1.amazonaws.com"
  )
)

Africa (Cape Town) : sts.af-south-1.amazonaws.com ; s3.af-south-1.amazonaws.com

Asia Pacific (Hong Kong) : sts.ap-east-1.amazonaws.com ; s3.ap-east-1.amazonaws.com

Europe (Milan) : sts.eu-south-1.amazonaws.com ; s3.eu-south-1.amazonaws.com

Middle East (Bahrain) : sts.me-south-1.amazonaws.com; s3.me-south-1.amazonaws.com

Re: Accessing the regions that are disabled by default in AWS from Databricks. In AWS we have 4 regi

AndreaCuda — Tue, 08 Apr 2025 13:46:34 GMT

Hello - We are looking to deploy and run Databricks in AWS in Bahrain, or UAE. Is this possible? This post is older so wondering if this is a viable option.

topic Re: Accessing the regions that are disabled by default in AWS from Databricks. In AWS we have 4 regi in Data Engineering

Accessing the regions that are disabled by default in AWS from Databricks. In AWS we have 4 regions that are disabled by default. You must first enabl...

Re: Accessing the regions that are disabled by default in AWS from Databricks. In AWS we have 4 regi