- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-19-2024 05:57 PM
I am encountering very similar behavior to drii_cavalcanti. When I use a Shared cluster with an IAM Role specified I can verify that the aws cli is installed but when I run aws sts get-caller-identity I receive the error "Unable to locate credentials. You can configure credentials by running "aws configure".". If I clone the cluster but change the type to No isolation shared it allows access. Based on some additional searching I think the real answer is https://community.databricks.com/t5/data-engineering/aws-secrets-works-in-one-cluster-but-not-another/m-p/68217/highlight/true#M33596 from a similar thread -- "
The limitation you’re encountering is related to the network and file system access of Shared Clusters. Specifically, Shared Clusters cannot connect to the Instance Metadata Service (IMDS), other EC2 instances, or any other services running within the Databricks VPC. This restriction prevents access to services that rely on the IMDS, including boto3 and the AWS CLI1."