Hey! So far I have followed along with the Configure S3 access with instance profiles article to grant my cluster access to an S3 bucket. I have also made sure to disable IAM role passthrough on the cluster.

Upon querying the bucket through a notebook using:

dbutils.fs.ls("s3://<bucket-name>/")

I receive a 403: Access denied message back from Amazon. I've double checked and AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are not present in the spark environment variables. I've also checked through this article - https://kb.databricks.com/en_US/security/forbidden-access-to-s3-data and have made sure to follow all the best practices.

Does anyone have any recommendations on what I should check or test?