cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

AWS Secrets Manager access

patrykp
New Contributor II

Hi

I am trying to establish a method of accessing secrets from AWS Secrets Manager and understand this can be done with boto as suggested from AWS.

We have created all of the relevant IAM roles, instance profiles etc.  Accessing S3 with this method is working well.

However, whenever I try to interact with Secrets Manager I keep getting the
"NoCredentialsError: Unable to locate credentials" error.

The role assigned to the EC2 after creation has the relevant permission to read/write from Secrets Manager.

I'm at a dead end and appreciate any help.

Many Thanks


Example code:

 

import boto3

 

        session = boto3.Session(region_name=region_name)
        # Create a Secrets Manager client using instance profile/IAM role
        client = session.client(service_name='secretsmanager')
       
        # Get the secret value
        response = client.get_secret_value(
            SecretId=secret_id
        )

 

2 REPLIES 2

Alberto_Umana
Databricks Employee
Databricks Employee

Hi @patrykp,

Verify that the attached IAM policy includes the necessary secretsmanager:GetSecretValue, secretsmanager:ListSecrets

patrykp
New Contributor II

Hi @Alberto_Umana,

Yes, the Role has the SecretsManagerReadWrite policy.

Also within my furthet investigation I tried running it via Personal Cluster and it worked!

Basically, 3 scenarios:

- Shared Cluster with applied InstanceProfile - Secrets failing
- Personal without InstanceProfile  applied - Secrets failing
- Shared Cluster with applied InstanceProfile - Secrets received succesfully.

So I guess problem is with the Access mode of the cluster? Cannot find anything related to that in the docs.

 

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now