topic The cluster with the instance profile cannot access the S3 bucket. 403 permission denied is thrown in Data Engineering https://community.databricks.com/t5/data-engineering/the-cluster-with-the-instance-profile-cannot-access-the-s3/m-p/21914#M14977 <P>The <A href="https://docs.databricks.com/administration-guide/cloud-configurations/aws/instance-profiles.html" alt="https://docs.databricks.com/administration-guide/cloud-configurations/aws/instance-profiles.html" target="_blank">document</A> has been followed to configure the instance profile. The ec2 instance is able to access the S3 bucket when configured the same instance profile. However, the cluster configured to use the same instance profile failed to access the S3 bucket due to permission denied. </P> Mon, 21 Jun 2021 23:05:50 GMT User16752239289 2021-06-21T23:05:50Z The cluster with the instance profile cannot access the S3 bucket. 403 permission denied is thrown https://community.databricks.com/t5/data-engineering/the-cluster-with-the-instance-profile-cannot-access-the-s3/m-p/21914#M14977 <P>The <A href="https://docs.databricks.com/administration-guide/cloud-configurations/aws/instance-profiles.html" alt="https://docs.databricks.com/administration-guide/cloud-configurations/aws/instance-profiles.html" target="_blank">document</A> has been followed to configure the instance profile. The ec2 instance is able to access the S3 bucket when configured the same instance profile. However, the cluster configured to use the same instance profile failed to access the S3 bucket due to permission denied. </P> Mon, 21 Jun 2021 23:05:50 GMT https://community.databricks.com/t5/data-engineering/the-cluster-with-the-instance-profile-cannot-access-the-s3/m-p/21914#M14977 User16752239289 2021-06-21T23:05:50Z Re: The cluster with the instance profile cannot access the S3 bucket. 403 permission denied is thrown https://community.databricks.com/t5/data-engineering/the-cluster-with-the-instance-profile-cannot-access-the-s3/m-p/21915#M14978 <P>I suspect this is due to AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY has been added to the spark environmental variable. </P><P></P><P>You can run %sh env | grep -i aws on your cluster and make sure AWS_ACCESS_KEY_ID is not present. </P><P></P><P>If so, then please remove it either from the cluster advanced options -&gt; spark -&gt; Environment Variables or from the init scripts. Please make sure check both cluster scope init script and global init script. </P> Mon, 21 Jun 2021 23:14:16 GMT https://community.databricks.com/t5/data-engineering/the-cluster-with-the-instance-profile-cannot-access-the-s3/m-p/21915#M14978 User16752239289 2021-06-21T23:14:16Z