DBUtils commands do not work on shared access mode clusters

[drii_cavalcanti]

Hi there,

I am trying to upload a file to an s3 bucket. However, none of dbutils commands seem to work neither does the boto3 library. For clusters that have the configuration, except for the shared access mode, seem to work fine.

Those are the error messages that I am getting:

java.nio.file.AccessDeniedException: : Instantiate shaded.databricks.org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: User: arn:aws:sts::*:assumed-role/[same as the instance profile config on the cluster]/i-* is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::*:role/[same as the instance profile config on the cluster] (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied)

java.nio.file.AccessDeniedException: s3://: shaded.databricks.org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by AwsCredentialContextTokenProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [com.databricks.backend.daemon.driver.aws.AwsLocalCredentialContextTokenProvider@*: No role specified and no roles available., com.databricks.backend.daemon.driver.aws.ProxiedIAMCredentialProvider@*: User does not have any IAM roles]

Unable to locate credentials. You can configure credentials by running "aws configure". (Even though, it is configured during the starting time by an init-scripts)

Has anyone encountered this issue before? If so, is there anything that I am missing here?

Thank you so much,
Adriana Cavalcanti

[drii_cavalcanti]

Hi @Retired_mod ,

Thanks for your patience and reply. However, I am unsure if I was not clear, but when I change the access mode to No Isolation Shared, the dbutils commands and AWS credentials work fine. However, they do not work on Shared Access Mode.

Adriana Cavalcanti

[mchugani]

@drii_cavalcanti Were you able to resolve this?

[mvdilts1]

I am encountering very similar behavior to drii_cavalcanti.  When I use a Shared cluster with an IAM Role specified I can verify that the aws cli is installed but when I run aws sts get-caller-identity I receive the error "Unable to locate credentials. You can configure credentials by running "aws configure".".  If I clone the cluster but change the type to No isolation shared it allows access.  Based on some additional searching I think the real answer is https://community.databricks.com/t5/data-engineering/aws-secrets-works-in-one-cluster-but-not-another/m-p/68217/highlight/true#M33596 from a similar thread --  "

The limitation you’re encountering is related to the network and file system access of Shared Clusters. Specifically, Shared Clusters cannot connect to the Instance Metadata Service (IMDS), other EC2 instances, or any other services running within the Databricks VPC. This restriction prevents access to services that rely on the IMDS, including boto3 and the AWS CLI1."