cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Getting file permission issues even though I have the right IAM role attached

User16869510359
Esteemed Contributor

I am reading data from S3 from a Databricks cluster and the read operation seldom fails with 403 permission errors. Restarting the cluster fixes my issue.

1 ACCEPTED SOLUTION

Accepted Solutions

User16869510359
Esteemed Contributor

The main reason for this behavior is :

  • AWS keys are used in addition to the IAM role. Using global init scripts to set the AWS keys can cause this behavior.
  • The IAM role has the required permission to access the S3 data, but AWS keys are set in the Spark configuration. For example, setting spark.hadoop.fs.s3a.secret.key can conflict with the IAM role.
  • Setting AWS keys at the environment level on the driver node from an interactive cluster through a notebook.

View solution in original post

1 REPLY 1

User16869510359
Esteemed Contributor

The main reason for this behavior is :

  • AWS keys are used in addition to the IAM role. Using global init scripts to set the AWS keys can cause this behavior.
  • The IAM role has the required permission to access the S3 data, but AWS keys are set in the Spark configuration. For example, setting spark.hadoop.fs.s3a.secret.key can conflict with the IAM role.
  • Setting AWS keys at the environment level on the driver node from an interactive cluster through a notebook.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.