cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

AWS Secrets Works In One Cluster But Not Another

dbdude
New Contributor II

Why can I use boto3 to go to secrets manager to retrieve a secret with a personal cluster but I get an error with a shared cluster?

NoCredentialsError: Unable to locate credentials

 

3 REPLIES 3

Szpila
New Contributor III

Nice reply from chat GPT, but it seems that the true cause is that Databricks intentionally prevent prevent users from using the credentials of the host machine.

drii_cavalcanti
New Contributor III

Hey @Szpila , have you found a solution for it? I am currently encountering the same issue.

Husky
New Contributor III

Hey @dbdude, I am facing the same error. Did you find a solution to access the AWS credentials on a Shared Cluster?

This article describes a way of storing credentials in a Unity Catalog Volume to fetch by the Shared Cluster:

https://medium.com/@amlucius/securely-accessing-external-services-on-databricks-shared-clusters-with...

But I am not a fan of storing the credentials in a Bucket..

@Kaniz_Fatma The reason why fetching the AWS credentials on a Shared Cluster does not work is a limitation of the network and file system access of Shared Clusters. See https://docs.databricks.com/en/compute/access-mode-limitations.html

Cannot connect to the instance metadata service (IMDS), other EC2 instances, or any other services running in the Databricks VPC. This prevents access to any service that uses the IMDS, such as boto3 and the AWS CLI.

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group