Hey @dbdude, I am facing the same error. Did you find a solution to access the AWS credentials on a Shared Cluster?
This article describes a way of storing credentials in a Unity Catalog Volume to fetch by the Shared Cluster:
https://medium.com/@amlucius/securely-accessing-external-services-on-databricks-shared-clusters-with...
But I am not a fan of storing the credentials in a Bucket..
@Kaniz_Fatma The reason why fetching the AWS credentials on a Shared Cluster does not work is a limitation of the network and file system access of Shared Clusters. See https://docs.databricks.com/en/compute/access-mode-limitations.html
Cannot connect to the instance metadata service (IMDS), other EC2 instances, or any other services running in the Databricks VPC. This prevents access to any service that uses the IMDS, such as boto3 and the AWS CLI.