topic Re: AWS Secrets Works In One Cluster But Not Another in Data Engineering

AWS Secrets Works In One Cluster But Not Another

dbdude — Thu, 17 Aug 2023 23:01:48 GMT

Why can I use boto3 to go to secrets manager to retrieve a secret with a personal cluster but I get an error with a shared cluster?

NoCredentialsError: Unable to locate credentials

Re: AWS Secrets Works In One Cluster But Not Another

Szpila — Fri, 10 Nov 2023 10:05:10 GMT

Nice reply from chat GPT, but it seems that the true cause is that Databricks intentionally prevent prevent users from using the credentials of the host machine.

Re: AWS Secrets Works In One Cluster But Not Another

drii_cavalcanti — Wed, 24 Jan 2024 04:56:57 GMT

Hey @Szpila , have you found a solution for it? I am currently encountering the same issue.

Re: AWS Secrets Works In One Cluster But Not Another

Husky — Mon, 06 May 2024 09:57:59 GMT

Hey @dbdude, I am facing the same error. Did you find a solution to access the AWS credentials on a Shared Cluster?

This article describes a way of storing credentials in a Unity Catalog Volume to fetch by the Shared Cluster:

https://medium.com/@amlucius/securely-accessing-external-services-on-databricks-shared-clusters-with-uc-volumes-9235616b9b02

But I am not a fan of storing the credentials in a Bucket..

@Retired_mod The reason why fetching the AWS credentials on a Shared Cluster does not work is a limitation of the network and file system access of Shared Clusters. See https://docs.databricks.com/en/compute/access-mode-limitations.html

Cannot connect to the instance metadata service (IMDS), other EC2 instances, or any other services running in the Databricks VPC. This prevents access to any service that uses the IMDS, such as boto3 and the AWS CLI.