Managing Secrets for Different Groups in a Databricks Workspace

[Direo]

Hi everyone,

I'm looking for some advice on how people are managing secrets within Databricks when you have different groups (or teams) in the same workspace, each requiring access to different sets of secrets.

Here’s the challenge:

• We have multiple groups within the same Databricks workspace, and each group needs different sets of secrets.
• Some groups or even individual users need specific secrets with tightly controlled access.

My Questions:

1. Do you create separate Azure Key Vaults for each group or user, and then integrate them into Databricks? Or,
2. Do you use Databricks-backed secret scopes with different permissions per group?
3. Is there a best practice to ensure security while maintaining flexibility?

Additionally, if anyone has automated this process, I’d love to hear how:

• Are you automating secret management using tools like Terraform, ARM templates, or the Databricks API?
• Any tips on managing secret scope permissions dynamically as teams and their access needs change?

Thanks!

[Walter_C]

Managing secrets within Databricks when you have different groups or teams in the same workspace can be approached in several ways, each with its own advantages. Here are some best practices and methods based on the context provided:

1. Using Azure Key Vaults:

   • Separate Key Vaults for Each Group/User: You can create separate Azure Key Vaults for each group or user and then integrate them into Databricks. This method allows for fine-grained access control and isolation of secrets. Each Key Vault can have its own access policies, ensuring that only the intended group or user can access the secrets.
   • Azure Key Vault-backed Secret Scopes: Databricks allows you to create secret scopes that are backed by Azure Key Vault. This means that secrets are stored in Azure Key Vault and accessed through Databricks. This method leverages Azure's robust security features and integrates seamlessly with Databricks.

2. Databricks-backed Secret Scopes:

   • Creating Secret Scopes with Different Permissions: You can create Databricks-backed secret scopes and assign different permissions to each group or user. This method is straightforward and allows you to manage secrets directly within Databricks. You can use the Databricks CLI or the Secrets API to create and manage these scopes.
   • Managing Permissions: By default, the user who creates the secret scope has the MANAGE permission, which allows them to read, write, and manage permissions on the scope. You can grant other users or groups specific permissions (READ, WRITE, MANAGE) on the secret scope using the Databricks CLI or Secrets API.

3. Automation of Secret Management:

   • Using Terraform: Terraform can be used to automate the creation and management of Azure Key Vaults, secret scopes, and access policies. This approach ensures that your infrastructure is defined as code and can be versioned and managed consistently.
   • ARM Templates: Azure Resource Manager (ARM) templates can also be used to automate the deployment and configuration of Azure Key Vaults and their access policies.
   • Databricks API: The Databricks API can be used to programmatically create and manage secret scopes and secrets. This can be integrated into your CI/CD pipelines to ensure that secrets are managed dynamically as teams and their access needs change