The main reason for this behavior is :

• AWS keys are used in addition to the IAM role. Using global init scripts to set the AWS keys can cause this behavior.
• The IAM role has the required permission to access the S3 data, but AWS keys are set in the Spark configuration. For example, setting spark.hadoop.fs.s3a.secret.key can conflict with the IAM role.
• Setting AWS keys at the environment level on the driver node from an interactive cluster through a notebook.