AWS Secrets Works In One Cluster But Not Another

[dbdude]

Why can I use boto3 to go to secrets manager to retrieve a secret with a personal cluster but I get an error with a shared cluster?

NoCredentialsError: Unable to locate credentials

[Szpila]

Nice reply from chat GPT, but it seems that the true cause is that Databricks intentionally prevent prevent users from using the credentials of the host machine.

[drii_cavalcanti]

Hey @Szpila , have you found a solution for it? I am currently encountering the same issue.

[Husky]

Hey @dbdude, I am facing the same error. Did you find a solution to access the AWS credentials on a Shared Cluster?

This article describes a way of storing credentials in a Unity Catalog Volume to fetch by the Shared Cluster:

https://medium.com/@amlucius/securely-accessing-external-services-on-databricks-shared-clusters-with...

But I am not a fan of storing the credentials in a Bucket..

@Retired_mod The reason why fetching the AWS credentials on a Shared Cluster does not work is a limitation of the network and file system access of Shared Clusters. See https://docs.databricks.com/en/compute/access-mode-limitations.html

Cannot connect to the instance metadata service (IMDS), other EC2 instances, or any other services running in the Databricks VPC. This prevents access to any service that uses the IMDS, such as boto3 and the AWS CLI.