Databricks Key Vault Secret - Is it available in Databricks on AWS?

[Ramana]

@Hubert-Dudek, I see your post regarding Key Value Secret handling via UI for Databricks on Azure.

Is this feature available for Databricks on AWS as well?

#Secrets #Scopes #DatabricksOnAWS

Thanks

Ramana

[lingareddy_Alva]

Hi @Ramana 

Yes, the Key-Value secret handling feature is available for Databricks on AWS as well! The Databricks Secret Management functionality works across all cloud platforms (AWS, Azure, and GCP) with very similar capabilities.

Databricks Secrets on AWS
Available Secret Scopes:
1. Databricks-backed scopes - Managed by Databricks itself
2. AWS Key Vault-backed scopes - Integrated with AWS Secrets Manager or AWS Systems Manager Parameter Store

This works the same way on AWS as it does on Azure - you can create, manage, and view secret scopes through the web interface.
The functionality is essentially identical across cloud platforms, with the main differences being in the backend integration options (AWS services vs Azure services vs GCP services).

[Ramana]

Is there a reference article from Databricks or anyone talking about it?

That would be helpful to share with my team.

[lingareddy_Alva]

Go through documents below.

Databricks Secret Management Guide:
https://docs.databricks.com/security/secrets/index.html

AWS-Specific Documentation:
Store secrets in AWS Secrets Manager:
https://docs.databricks.com/en/security/secrets/aws-secrets-manager.html

Secret scope backed by AWS Secrets Manager: This is typically covered within the main secrets documentation

Databricks CLI for Secrets:
Databricks CLI Secrets Commands: https://docs.databricks.com/dev-tools/cli/secrets-cli.html

CLI Installation and Setup: https://docs.databricks.com/dev-tools/cli/index.html

[Ramana]

Currently, we manage Scopes and Secrets via CLI, API, and Terraform.

Today, I read an article about UC to Azure Key-Vault Integration written by @Hubert-Dudek, I thought that there would be a similar option for AWS as well. But it looks like it is not available for Databricks on AWS.

Here is the one I am referring to: https://databrickster.medium.com/unity-catalog-to-azure-key-vault-no-more-dbutils-secrets-0a8248af46...

FYI: https://docs.databricks.com/en/security/secrets/aws-secrets-manager.html This link is broken.

We also use AWS Secret Manager, but I don't think there is a tight integration between AWS and Databricks like Azure and Databricks for Key Vault.

If you know or see any article related to the one I provided, please share. That would be appreciated.

[lingareddy_Alva]

I agree. Sure, I will share if I come across. Thanks.

[ceceliac]

Hi, do you have any update on this topic? We are looking for the same thing.  We are using Databricks (UC) on AWS and have a developer group that wants to implement 90-day rotation for an AWS IAM secret using Secrets Manager.  I do not see anything in the documentation indicating that this is supported / possible.