Secret scope with Azure RBAC

[achistef]

Hello!

We have lots of Azure keyvaults that we use in our Azure Databricks workspaces. We have created secret scopes that are backed by the keyvaults. Azure supports two ways of authenticating to keyvaults:

- Access policies, which has been marked as legacy.

- Role-based access control (RBAC), which is a unified standard in all Azure services.

The Databricks secret scopes can only be defined using access policies. Given that Azure stirs towards RBAC, is there a plan to support RBAC for secret scopes?

[daniel_sahal · Accepted solution]

@achistef 

Actually, RBAC is supported for authentication for the secret scopes.

The thing is, when you setup the secret scope, Databricks is automatically assigning permissions through access policies. With RBAC - you'll need to grant the role on your own.

As a test:

1. I've created an Azure KeyVault with "Azure role-based access control" as a permission model.
2. Navigated to https://<databricks-instance>#secrets/createScope and created a secret scope
3. In Azure KeyVault IAM, added Key Vault Secrets User to the AzureDatabricks (fa5c679a-c02e-4f33-a397-7419315171b3) Application
4. Created a test secret and tried to access that from a notebook. Tada, it works.

[szymon_dybczak]

Hi @achistef ,

As Daniel mentioned, RBAC is supported, but you should be aware of the consequences it entails.
For Datbricks to connect to Keyvault on RBAC we add AzureDatabricks Enterprise Application ID in RBAC, but this allows all the Databricks instances deployed in that tenant to have an access to that KeyVault.
You can read more in below discussion:

Allow only a specific Azure Databricks instance to connect to keyvault - Microsoft Q&A

And here is video how to configure Databricks and KeyVault using RBAC:

Unlocking Secrets in Azure Databricks with Azure Key Vault! 🗝️✨ | Azure Databricks Tutorials (youtu...

[daniel_sahal]

@szymon_dybczak When using Access Policies, you're still adding the permissions to AzureDatabricks SP, so it's kinda the same issue as with RBAC. That's why I'm not a big fan of having secret scopes at all.

What's more, to even create a secret scope in Databricks, you need (i mean, a user who creates a secret scope) a Contributor or Owner role on the KeyVault, so that's a little bit of security that was added here.

[szymon_dybczak]

@daniel_sahal  I'm not a fan of this solution either. The worst part is that neither Databricks nor Microsoft want to address this issue. And it's been known for years...
From security perspective, I think it's better to handle secrets by yourself, i.e writing custom library. That way you have much more granular control over who has access to what.

[achistef]

That is very helpful, thank you for your answers.

FYI there is some outdated documentation about this topic 
https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes#configure-your-azu...

[Chamak]

@daniel_sahal Where in the portal did you get the App ID for AzureDatabricks (fa5c679a-c02e-4f33-a397-7419315171b3) Application? I can't seem to find it. Thanks.

[kuldeep-in]

@Chamak You can find 'AzureDatabricks' in User, group or service principal assignment. You dont need to find application id, as it will automatically displayed when you add AzureDatabricks as member.

cc: @daniel_sahal