Managing Secrets for Different Groups in a Databricks Workspace

[Direo]

Hi everyone,

I'm looking for some advice on how people are managing secrets within Databricks when you have different groups (or teams) in the same workspace, each requiring access to different sets of secrets.

Here’s the challenge:

• We have multiple groups within the same Databricks workspace, and each group needs different sets of secrets.
• Some groups or even individual users need specific secrets with tightly controlled access.

My Questions:

1. Do you create separate Azure Key Vaults for each group or user, and then integrate them into Databricks? Or,
2. Do you use Databricks-backed secret scopes with different permissions per group?
3. Is there a best practice to ensure security while maintaining flexibility?

Additionally, if anyone has automated this process, I’d love to hear how:

• Are you automating secret management using tools like Terraform, ARM templates, or the Databricks API?
• Any tips on managing secret scope permissions dynamically as teams and their access needs change?

Thanks!