hey @messiah 

Probably the problem is in the cluster configuration

When using a Shared Cluster in Databricks, the Instance Profile assigned to the cluster will not be used to authenticate access to AWS resources like S3. This is because Shared Clusters operate in a multi-user environment, where permissions and credentials are tied to the individual user rather than the cluster itself.

To work around this limitation, the best approach is to use External Locations if your workspace is enabled for Unity Catalog. External Locations allow administrators to define and manage access to cloud storage at the Unity Catalog level, ensuring that users can read and write data securely without needing direct access to AWS credentials.

If Unity Catalog is not available or External Locations are not an option, a simple and effective alternative is to use a Single-User Cluster instead of a Shared Cluster. Single-User Clusters operate in an isolated environment where all commands run under the same user identity, allowing the assigned Instance Profile to be applied correctly. This means the cluster will have seamless access to AWS resources without requiring additional authentication mechanisms.

By leveraging External Locations where possible or switching to Single-User Clusters, you can avoid authentication issues while ensuring secure and efficient access to cloud storage in Databricks.

Hope that helps 🙂