topic Re: Allow read access to S3 buckets from one AWS accounts to other AWS accounts. in Data Engineering

Allow read access to S3 buckets from one AWS accounts to other AWS accounts.

164079 — Wed, 28 Dec 2022 14:11:40 GMT

Dear team,

We have several AWS accounts with S3 buckets, the databricks setup is on our dev AWS account and we would like to allow instance profile to have read permission on all our S3 buckets on the other AWS accounts ( without using bucket policy which require us to add it on any bucket)

I am trying using the assume role but dosent work, getting access denied.

It is working only if i set S3 bucket permissions on my other/remote AWS account bucket policy.

Please advise

Thansks!

Re: Allow read access to S3 buckets from one AWS accounts to other AWS accounts.

User16255483290 — Wed, 28 Dec 2022 19:09:37 GMT

Can you please share the IAM role policy in the secondary account (Bucket account) ?

Just wanted to know have you tried setting the config in the cluster.

spark.hadoop.fs.s3a.bucket.<s3-bucket-name>.aws.credentials.provider org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider

spark.hadoop.fs.s3a.bucket.<s3-bucket-name>.assumed.role.arn arn:aws:iam::<bucket-owner-account-id>:role/Master_Role

Re: Allow read access to S3 buckets from one AWS accounts to other AWS accounts.

164079 — Thu, 29 Dec 2022 08:59:48 GMT

Thank you @D Raj Kumar

Added it and now its works!

Thanks