Not able to read S3 object through AssumedRoleCredentialProvider

[Sweetnesh]

SparkSession spark = SparkSession.builder()

.appName("SparkS3Example")

.master("local[1]")

.getOrCreate();

spark.sparkContext().hadoopConfiguration().set("fs.s3a.access.key", S3_ACCOUNT_KEY);

spark.sparkContext().hadoopConfiguration().set("fs.s3a.secret.key", S3_SECRET_KEY);

spark.sparkContext().hadoopConfiguration().set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider");

spark.sparkContext().hadoopConfiguration().set("fs.s3a.assumed.role.credentials.provider", "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider");

spark.sparkContext().hadoopConfiguration().set("fs.s3a.assumed.role.arn", "arn:aws:iam::000000000000:role/<bucket>");

spark.sparkContext().hadoopConfiguration().set("fs.s3a.assumed.role.sts.endpoint.region", awsRegion);

spark.sparkContext().hadoopConfiguration().set("fs.s3a.assumed.role.session.name", "test");

spark.sparkContext().hadoopConfiguration().set("fs.s3a.assumed.role.sts.endpoint", "sts.ap-southeast-2.amazonaws.com");

spark.sparkContext().hadoopConfiguration().set("fs.s3a.endpoint", "s3.ap-southeast-2.amazonaws.com");

spark.sparkContext().hadoopConfiguration().set("fs.s3a.endpoint.region", awsRegion

Able to get AccessKeyId, SecretAccessKey and SessionToken through AssumedRoleCredentialProvider, but not able to get object from S3, it is throwing 403 AccessDenied exception.

But If I used same AccessKeyId, SecretAccessKey and SessionToken in AWS CLI then able to download using cp command through command prompt.

Expected to access S3 object through AssumedRoleCredentialProvider.

[Debayan]

Similar thread: https://repost.aws/knowledge-center/s3-troubleshoot-403

[Vartika]

Hi @Sweetnesh Dholariya​,

Does @Debayan Mukherjee​'s response answer your question? If yes, would you be happy to mark it as best so that other members can find the solution more quickly?

Thanks!