Databricks AWS Secrets Manager access

[eric-cordeiro]

I have a workspace deployed in AWS and need to read some secrets from AWS Secrets Manager in my notebook. I'm aware that there is no default process similar to Azure Key Vault, however I know that we can try to access it using boto3, but I'm stuck at the authentication process. I'm not allowed by the company to create key values for it, so it needs to be done with roles. I've been trying to follow the Databricks documentation to use instance profiles in the cluster (https://docs.databricks.com/en/aws/iam/instance-profile-tutorial.html) but it's specific for s3 and not necessarily for AWS secrets manager. If someone have done it before and could share how is the appropriated way, it would be very appreciated!

[eric-cordeiro]

Thank you for the reply Kaniz ! By any chance, would you have any terraform reference to create, configure and attach this EC2 IAM role? 

[fbuechel92]

Hi @Retired_mod ,

I tried out your steps but it still doesn't work for me, possibly because I skipped step 3. That's because I already have a credential configuration for my workspace. So I just went on with step 4 and added what you described to the ec2 policy of the role which is linked in my current credential config.

Anyways, my question to you is whether you know of any documentation on this topic? I haven't found much online surprisingly.

Many thanks!

Fabian