Insufficient Permissions Error When Reading Data from S3 in Shared Databricks Compute

[vidya_kothavale]

I am using a Shared Databricks Compute and trying to read data from an S3 bucket via an Instance Profile. However, I am encountering the following error:

[INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission SELECT on any file. SQLSTATE: 42501

My Setup:

I am a Databricks workspace user.

The Instance Profile is correctly configured, and I can see it while creating the compute.

The S3 bucket has the necessary permissions for the instance profile.

Troubleshooting Steps Tried:

1. Verified that the instance profile is attached to the shared compute.

2. Checked S3 bucket permissions (s3:GetObject, s3:ListBucket).

3. Tried using a single-user cluster instead of shared compute (its working fine with single user compute).

Is this the expected behavior or a limitation of Shared Databricks Compute?

[Ayushi_Suthar · Accepted solution]

Hi @vidya_kothavale , Greetings!

Can you please refer to this article and check if it helps you to resolve your issue : https://kb.databricks.com/en_US/data/user-does-not-have-permission-select-on-any-file

Please note that these permissions are only required for a shared cluster. The security implications of granting ANY FILE permissions on a filesystem. You should only grant ANY FILE to privileged users. Users with lower privileges on the cluster should never access data by referencing an actual storage location. Instead, they should access data from tables that are created by privileged users, thus ensuring that Table ACLS are enforced.

Please let me know if this helps and leave a like if this information is useful, followups are appreciated.
Kudos
Ayushi