AWS Secrets Manager access

[AClarkson]

Hi

We're in the process of moving over from Databricks in Azure to AWS.

I am trying to establish a method of accessing secrets from AWS Secrets Manager (we were using Azure KeyVault) and understand this can be done with boto as suggested from AWS.

We have created all of the relevant IAM roles, instance profiles etc.  Accessing S3 with this method is working OK.

However, whenever I try to interact with Secrets Manager I keep getting the "NoCredentialsError: Unable to locate credentials" error.

The role assigned to the EC2 after creation has the relevant permission to read/write from Secrets Manager.

I'm at a dead end and appreciate any help.

Many Thanks

Example code:

import botocore

import botocore.session

from aws_secretsmanager_caching import SecretCache, SecretCacheConfig

client = botocore.session.get_session().create_client('secretsmanager', region_name='aws-region')

cache_config = SecretCacheConfig()

cache = SecretCache( config = cache_config, client = client)

secret = cache.get_secret_string('secretname')

[AClarkson]

I should add I'm running this on a shared cluster and therefore suspect the following constraints are impeding me:

Taken from:  https://docs.databricks.com/en/clusters/configure.html

• Cannot connect to the instance metadata service (IMDS), other EC2 instances, or any other services running in the Databricks VPC.

  • This prevents access to any service that uses the IMDS, such as boto3 and the AWS CLI.

So begs the question, how do I interface with AWS Secrets Manager from a shared cluster ?

Thanks!

[Kaniz · Accepted solution]

Hi @AClarkson, I understand that you are trying to access AWS Secrets Manager from a shared cluster and are facing issues due to the constraints mentioned in the documentation . Since the instance metadata service (IMDS) is not accessible from the shared cluster, you cannot use the AWS SDK to retrieve temporary security credentials .

However, there are a few workarounds that you can try to access AWS Secrets Manager from a shared cluster:

1. You can create an IAM user with the necessary permissions to access AWS Secrets Manager and then use the access key ID and secret access key of the IAM user to authenticate your requests . You can then store the access key ID and secret access key in a configuration file or environment variables on the shared cluster.
2. You can use the AWS Systems Manager Parameter Store to store your secrets instead of AWS Secrets Manager . The AWS Systems Manager Parameter Store does not require access to the IMDS and can be accessed from the shared cluster.
3. You can use a third-party secrets management service such as HashiCorp Vault or CyberArk to store your secrets . These services provide a secure way to store and manage secrets and can be accessed from the shared cluster.

I hope this helps! Let me know if you have any other questions.

[AClarkson]

Thanks Kaniz for your comprehensive response.

We are going to use the Databricks secrets for the time being.

[Kaniz]

Awesome, Thank you! Happy Learning!