08-30-2024 04:03 AM
Hello!
We have lots of Azure keyvaults that we use in our Azure Databricks workspaces. We have created secret scopes that are backed by the keyvaults. Azure supports two ways of authenticating to keyvaults:
- Access policies, which has been marked as legacy.
- Role-based access control (RBAC), which is a unified standard in all Azure services.
The Databricks secret scopes can only be defined using access policies. Given that Azure stirs towards RBAC, is there a plan to support RBAC for secret scopes?
09-01-2024 11:29 PM
Actually, RBAC is supported for authentication for the secret scopes.
The thing is, when you setup the secret scope, Databricks is automatically assigning permissions through access policies. With RBAC - you'll need to grant the role on your own.
As a test:
1. I've created an Azure KeyVault with "Azure role-based access control" as a permission model.
2. Navigated to https://<databricks-instance>#secrets/createScope and created a secret scope
3. In Azure KeyVault IAM, added Key Vault Secrets User to the AzureDatabricks (fa5c679a-c02e-4f33-a397-7419315171b3) Application
4. Created a test secret and tried to access that from a notebook. Tada, it works.
09-01-2024 11:29 PM
Actually, RBAC is supported for authentication for the secret scopes.
The thing is, when you setup the secret scope, Databricks is automatically assigning permissions through access policies. With RBAC - you'll need to grant the role on your own.
As a test:
1. I've created an Azure KeyVault with "Azure role-based access control" as a permission model.
2. Navigated to https://<databricks-instance>#secrets/createScope and created a secret scope
3. In Azure KeyVault IAM, added Key Vault Secrets User to the AzureDatabricks (fa5c679a-c02e-4f33-a397-7419315171b3) Application
4. Created a test secret and tried to access that from a notebook. Tada, it works.
09-01-2024 11:57 PM - edited 09-01-2024 11:59 PM
Hi @achistef ,
As Daniel mentioned, RBAC is supported, but you should be aware of the consequences it entails.
For Datbricks to connect to Keyvault on RBAC we add AzureDatabricks Enterprise Application ID in RBAC, but this allows all the Databricks instances deployed in that tenant to have an access to that KeyVault.
You can read more in below discussion:
Allow only a specific Azure Databricks instance to connect to keyvault - Microsoft Q&A
And here is video how to configure Databricks and KeyVault using RBAC:
Unlocking Secrets in Azure Databricks with Azure Key Vault! 🗝️✨ | Azure Databricks Tutorials (youtu...
09-02-2024 11:59 PM
@szymon_dybczak When using Access Policies, you're still adding the permissions to AzureDatabricks SP, so it's kinda the same issue as with RBAC. That's why I'm not a big fan of having secret scopes at all.
What's more, to even create a secret scope in Databricks, you need (i mean, a user who creates a secret scope) a Contributor or Owner role on the KeyVault, so that's a little bit of security that was added here.
09-03-2024 02:41 AM
@daniel_sahal I'm not a fan of this solution either. The worst part is that neither Databricks nor Microsoft want to address this issue. And it's been known for years...
From security perspective, I think it's better to handle secrets by yourself, i.e writing custom library. That way you have much more granular control over who has access to what.
09-02-2024 10:29 PM
That is very helpful, thank you for your answers.
FYI there is some outdated documentation about this topic
https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes#configure-your-azu...
4 weeks ago
@daniel_sahal Where in the portal did you get the App ID for AzureDatabricks (fa5c679a-c02e-4f33-a397-7419315171b3) Application? I can't seem to find it. Thanks.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.
If there isn’t a group near you, start one and help create a community that brings people together.
Request a New Group