https://community.databricks.com/t5/data-engineering/getting-file-permission-issues-even-though-i-have-the-right-iam/m-p/19918#M13424 <P>I am reading data from S3 from a Databricks cluster and the read operation seldom fails with 403 permission errors. Restarting the cluster fixes my issue. </P> Fri, 25 Jun 2021 12:54:16 GMT brickster_2018 2021-06-25T12:54:16Z https://community.databricks.com/t5/data-engineering/getting-file-permission-issues-even-though-i-have-the-right-iam/m-p/19918#M13424 <P>I am reading data from S3 from a Databricks cluster and the read operation seldom fails with 403 permission errors. Restarting the cluster fixes my issue. </P> Fri, 25 Jun 2021 12:54:16 GMT https://community.databricks.com/t5/data-engineering/getting-file-permission-issues-even-though-i-have-the-right-iam/m-p/19918#M13424 brickster_2018 2021-06-25T12:54:16Z https://community.databricks.com/t5/data-engineering/getting-file-permission-issues-even-though-i-have-the-right-iam/m-p/19919#M13425 <P>The main reason for this behavior is : </P><UL><LI>AWS keys are used in addition to the IAM role. Using global init scripts to set the AWS keys can cause this behavior.</LI><LI>The IAM role has the required permission to access the S3 data, but AWS keys are set in the Spark configuration. For example, setting&nbsp;spark.hadoop.fs.s3a.secret.key&nbsp;can conflict with the IAM role.</LI><LI>Setting AWS keys at the environment level on the driver node from an interactive cluster through a notebook.</LI></UL><P></P> Fri, 25 Jun 2021 12:56:18 GMT https://community.databricks.com/t5/data-engineering/getting-file-permission-issues-even-though-i-have-the-right-iam/m-p/19919#M13425 brickster_2018 2021-06-25T12:56:18Z