AWS Secrets Manager access

[AClarkson]

Hi

We're in the process of moving over from Databricks in Azure to AWS.

I am trying to establish a method of accessing secrets from AWS Secrets Manager (we were using Azure KeyVault) and understand this can be done with boto as suggested from AWS.

We have created all of the relevant IAM roles, instance profiles etc.  Accessing S3 with this method is working OK.

However, whenever I try to interact with Secrets Manager I keep getting the "NoCredentialsError: Unable to locate credentials" error.

The role assigned to the EC2 after creation has the relevant permission to read/write from Secrets Manager.

I'm at a dead end and appreciate any help.

Many Thanks

Example code:

import botocore

import botocore.session

from aws_secretsmanager_caching import SecretCache, SecretCacheConfig

client = botocore.session.get_session().create_client('secretsmanager', region_name='aws-region')

cache_config = SecretCacheConfig()

cache = SecretCache( config = cache_config, client = client)

secret = cache.get_secret_string('secretname')

[AClarkson]

I should add I'm running this on a shared cluster and therefore suspect the following constraints are impeding me:

Taken from:  https://docs.databricks.com/en/clusters/configure.html

• Cannot connect to the instance metadata service (IMDS), other EC2 instances, or any other services running in the Databricks VPC.

  • This prevents access to any service that uses the IMDS, such as boto3 and the AWS CLI.

So begs the question, how do I interface with AWS Secrets Manager from a shared cluster ?

Thanks!

[AClarkson]

Thanks Kaniz for your comprehensive response.

We are going to use the Databricks secrets for the time being.

[heiko_u]

Hi,
Thank you for this suggestion. I am trying to implement this on DBR14.3 using the AWS Systems Manager Parameter Store, does require authentication. I tried both using the instance profile as well as environment variables, but neither work:

1) The instance profile approach seems to cause the same issue for AWS Systems Manager Parameter Store as when using AWS Secrets Manager (same IMDS problem)?

2) Storing "the access key ID and secret access key in a configuration file or environment variables" is not propagating to the workers or executors. I tried setting these both as cluster environment variables as well as in the spark config, but that did not work. Side note: I do not like this approach since I am not making credentials of one single service account accessible to "persons" using the databricks cluster. The instance profile seems a much more secure approach.

How can I make this work? We are using a shared UC enabled cluster on DBR 14.3

Best,

Heiko