DBUtils commands do not work on shared access mode clusters

[drii_cavalcanti]

Hi there,

I am trying to upload a file to an s3 bucket. However, none of dbutils commands seem to work neither does the boto3 library. For clusters that have the configuration, except for the shared access mode, seem to work fine.

Those are the error messages that I am getting:

java.nio.file.AccessDeniedException: : Instantiate shaded.databricks.org.apache.hadoop.fs.s3a.auth.AssumedRoleCredentialProvider: com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: User: arn:aws:sts::*:assumed-role/[same as the instance profile config on the cluster]/i-* is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::*:role/[same as the instance profile config on the cluster] (Service: AWSSecurityTokenService; Status Code: 403; Error Code: AccessDenied)

java.nio.file.AccessDeniedException: s3://: shaded.databricks.org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by AwsCredentialContextTokenProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [com.databricks.backend.daemon.driver.aws.AwsLocalCredentialContextTokenProvider@*: No role specified and no roles available., com.databricks.backend.daemon.driver.aws.ProxiedIAMCredentialProvider@*: User does not have any IAM roles]

Unable to locate credentials. You can configure credentials by running "aws configure". (Even though, it is configured during the starting time by an init-scripts)

Has anyone encountered this issue before? If so, is there anything that I am missing here?

Thank you so much,
Adriana Cavalcanti

[Kaniz]

Hi @drii_cavalcanti, The error messages you’re seeing suggest a few potential issues:

Access Denied Error: This error typically occurs when the IAM role associated with your EC2 instance.... You might want to check the IAM role’s policy to ensure it has the necessary permissions.

No AWS Credentials Provided: This error suggests that the AWS SDK is unable to find valid authentica.... You can set these credentials in several ways, including environment variables (AWS_ACCESS_KEY_ID an....

Unable to locate credentials: This error message suggests that the AWS CLI is unable to find your credentials. You can configure these credentials by running aws configure in your terminal.

Regarding the dbutils commands, there could be several reasons why they are not working. One possibility is that the commands are not being recognized due to an issue with the path or envir.... Another possibility is that there might be some compatibility issues with the version of Databricks ....

For the boto3 library, make sure it is properly installed and imported. If you have multiple versions of Python installed, ensure that boto3 is installed for the correct ve....

[drii_cavalcanti]

Hi @Kaniz ,

Thanks for your patience and reply. However, I am unsure if I was not clear, but when I change the access mode to No Isolation Shared, the dbutils commands and AWS credentials work fine. However, they do not work on Shared Access Mode.

Adriana Cavalcanti