topic Re: Secret scope with Azure RBAC in Administration & Architecture

Secret scope with Azure RBAC

achistef — Fri, 30 Aug 2024 11:03:20 GMT

Hello!

We have lots of Azure keyvaults that we use in our Azure Databricks workspaces. We have created secret scopes that are backed by the keyvaults. Azure supports two ways of authenticating to keyvaults:

- Access policies, which has been marked as legacy.

- Role-based access control (RBAC), which is a unified standard in all Azure services.

The Databricks secret scopes can only be defined using access policies. Given that Azure stirs towards RBAC, is there a plan to support RBAC for secret scopes?

Re: Secret scope with Azure RBAC

daniel_sahal — Mon, 02 Sep 2024 06:29:07 GMT

@achistef

Actually, RBAC is supported for authentication for the secret scopes.

The thing is, when you setup the secret scope, Databricks is automatically assigning permissions through access policies. With RBAC - you'll need to grant the role on your own.

As a test:

1. I've created an Azure KeyVault with "Azure role-based access control" as a permission model.
2. Navigated to https://<databricks-instance>#secrets/createScope and created a secret scope
3. In Azure KeyVault IAM, added Key Vault Secrets User to the AzureDatabricks (fa5c679a-c02e-4f33-a397-7419315171b3) Application
4. Created a test secret and tried to access that from a notebook. Tada, it works.

Re: Secret scope with Azure RBAC

szymon_dybczak — Mon, 02 Sep 2024 06:59:23 GMT

Hi @achistef ,

As Daniel mentioned, RBAC is supported, but you should be aware of the consequences it entails.
For Datbricks to connect to Keyvault on RBAC we add AzureDatabricks Enterprise Application ID in RBAC, but this allows all the Databricks instances deployed in that tenant to have an access to that KeyVault.
You can read more in below discussion:

Allow only a specific Azure Databricks instance to connect to keyvault - Microsoft Q&A

And here is video how to configure Databricks and KeyVault using RBAC:

Unlocking Secrets in Azure Databricks with Azure Key Vault! 🗝️✨ | Azure Databricks Tutorials (youtube.com)

Re: Secret scope with Azure RBAC

achistef — Tue, 03 Sep 2024 05:29:50 GMT

That is very helpful, thank you for your answers.

FYI there is some outdated documentation about this topic
https://learn.microsoft.com/en-us/azure/databricks/security/secrets/secret-scopes#configure-your-azure-key-vault-instance-for-azure-databricks

Re: Secret scope with Azure RBAC

daniel_sahal — Tue, 03 Sep 2024 06:59:34 GMT

@szymon_dybczak When using Access Policies, you're still adding the permissions to AzureDatabricks SP, so it's kinda the same issue as with RBAC. That's why I'm not a big fan of having secret scopes at all.

What's more, to even create a secret scope in Databricks, you need (i mean, a user who creates a secret scope) a Contributor or Owner role on the KeyVault, so that's a little bit of security that was added here.

Re: Secret scope with Azure RBAC

szymon_dybczak — Tue, 03 Sep 2024 09:41:44 GMT

@daniel_sahal I'm not a fan of this solution either. The worst part is that neither Databricks nor Microsoft want to address this issue. And it's been known for years...
From security perspective, I think it's better to handle secrets by yourself, i.e writing custom library. That way you have much more granular control over who has access to what.

Re: Secret scope with Azure RBAC

Chamak — Tue, 15 Oct 2024 06:02:43 GMT

@daniel_sahal Where in the portal did you get the App ID for AzureDatabricks (fa5c679a-c02e-4f33-a397-7419315171b3) Application? I can't seem to find it. Thanks.

Re: Secret scope with Azure RBAC

kuldeep-in — Sun, 01 Dec 2024 19:47:59 GMT

@Chamak You can find 'AzureDatabricks' in User, group or service principal assignment. You dont need to find application id, as it will automatically displayed when you add AzureDatabricks as member.

cc: @daniel_sahal