Not able to read S3 object through AssumedRoleCredentialProvider

Sweetnesh — Mon, 29 May 2023 11:50:32 GMT

SparkSession spark = SparkSession.builder()

.appName("SparkS3Example")

.master("local[1]")

.getOrCreate();

spark.sparkContext().hadoopConfiguration().set("fs.s3a.access.key", S3_ACCOUNT_KEY);

spark.sparkContext().hadoopConfiguration().set("fs.s3a.secret.key", S3_SECRET_KEY);

spark.sparkContext().hadoopConfiguration().set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider");

spark.sparkContext().hadoopConfiguration().set("fs.s3a.assumed.role.credentials.provider", "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider");

spark.sparkContext().hadoopConfiguration().set("fs.s3a.assumed.role.arn", "arn:aws:iam::000000000000:role/<bucket>");

spark.sparkContext().hadoopConfiguration().set("fs.s3a.assumed.role.sts.endpoint.region", awsRegion);

spark.sparkContext().hadoopConfiguration().set("fs.s3a.assumed.role.session.name", "test");

spark.sparkContext().hadoopConfiguration().set("fs.s3a.assumed.role.sts.endpoint", "sts.ap-southeast-2.amazonaws.com");

spark.sparkContext().hadoopConfiguration().set("fs.s3a.endpoint", "s3.ap-southeast-2.amazonaws.com");

spark.sparkContext().hadoopConfiguration().set("fs.s3a.endpoint.region", awsRegion

Able to get AccessKeyId, SecretAccessKey and SessionToken through AssumedRoleCredentialProvider, but not able to get object from S3, it is throwing 403 AccessDenied exception.

But If I used same AccessKeyId, SecretAccessKey and SessionToken in AWS CLI then able to download using cp command through command prompt.

Expected to access S3 object through AssumedRoleCredentialProvider.

Re: Not able to read S3 object through AssumedRoleCredentialProvider

Vartika — Tue, 30 May 2023 08:52:57 GMT

Hi @Sweetnesh Dholariya,

Does @Debayan Mukherjee's response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

Thanks!

Re: Not able to read S3 object through AssumedRoleCredentialProvider

Debayan — Tue, 30 May 2023 07:13:28 GMT

topic Not able to read S3 object through AssumedRoleCredentialProvider in Data Engineering

Not able to read S3 object through AssumedRoleCredentialProvider

Re: Not able to read S3 object through AssumedRoleCredentialProvider

Re: Not able to read S3 object through AssumedRoleCredentialProvider