SparkSession spark = SparkSession.builder()
.appName("SparkS3Example")
.master("local[1]")
.getOrCreate();
spark.sparkContext().hadoopConfiguration().set("fs.s3a.access.key", S3_ACCOUNT_KEY);
spark.sparkContext().hadoopConfiguration().set("fs.s3a.secret.key", S3_SECRET_KEY);
spark.sparkContext().hadoopConfiguration().set("fs.s3a.aws.credentials.provider", "org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider");
spark.sparkContext().hadoopConfiguration().set("fs.s3a.assumed.role.credentials.provider", "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider");
spark.sparkContext().hadoopConfiguration().set("fs.s3a.assumed.role.arn", "arn:aws:iam::000000000000:role/<bucket>");
spark.sparkContext().hadoopConfiguration().set("fs.s3a.assumed.role.sts.endpoint.region", awsRegion);
spark.sparkContext().hadoopConfiguration().set("fs.s3a.assumed.role.session.name", "test");
spark.sparkContext().hadoopConfiguration().set("fs.s3a.assumed.role.sts.endpoint", "sts.ap-southeast-2.amazonaws.com");
spark.sparkContext().hadoopConfiguration().set("fs.s3a.endpoint", "s3.ap-southeast-2.amazonaws.com");
spark.sparkContext().hadoopConfiguration().set("fs.s3a.endpoint.region", awsRegion
Able to get AccessKeyId, SecretAccessKey and SessionToken through AssumedRoleCredentialProvider, but not able to get object from S3, it is throwing 403 AccessDenied exception.
But If I used same AccessKeyId, SecretAccessKey and SessionToken in AWS CLI then able to download using cp command through command prompt.
Expected to access S3 object through AssumedRoleCredentialProvider.
