Databricks

eric-cordeiro · ‎10-03-2023

I have a workspace deployed in AWS and need to read some secrets from AWS Secrets Manager in my notebook. I'm aware that there is no default process similar to Azure Key Vault, however I know that we can try to access it using boto3, but I'm stuck at the authentication process. I'm not allowed by the company to create key values for it, so it needs to be done with roles. I've been trying to follow the Databricks documentation to use instance profiles in the cluster (https://docs.databricks.com/en/aws/iam/instance-profile-tutorial.html) but it's specific for s3 and not necessarily for AWS secrets manager. If someone have done it before and could share how is the appropriated way, it would be very appreciated!

Kaniz · ‎10-04-2023

Hi @eric-cordeiro,

IAM roles are used for authentication to access AWS Secrets Manager from a Databricks Notebook in AWS.

- Create a Cross-Account IAM Role with permissions to access secrets in AWS Secrets Manager.
- Create an access policy that grants necessary permissions to the IAM role.
- Call the Create credential configuration API with the IAM role ARN to establish cross-account trust and get a reference ID.
- Add the IAM role to the EC2 instance policy attached to the Databricks workspace to allow it to assume the role and access secrets.
- Access AWS Secrets Manager using the provided Python code snippet using boto3.
- The code assumes the IAM role attached to the EC2 instance has the necessary permissions.

eric-cordeiro · ‎10-04-2023

Thank you for the reply Kaniz ! By any chance, would you have any terraform reference to create, configure and attach this EC2 IAM role?

fbuechel92 · ‎03-03-2024

Hi @Kaniz ,

I tried out your steps but it still doesn't work for me, possibly because I skipped step 3. That's because I already have a credential configuration for my workspace. So I just went on with step 4 and added what you described to the ec2 policy of the role which is linked in my current credential config.

Anyways, my question to you is whether you know of any documentation on this topic? I haven't found much online surprisingly.

Many thanks!

Fabian