topic Re: Can we use "Access Connector for Azure Databricks" to access Azure Key Vault? in Data Engineering

Can we use "Access Connector for Azure Databricks" to access Azure Key Vault?

grazie — Tue, 06 Dec 2022 14:24:53 GMT

We have a scenario where ideally we'd like to use Managed Identities to access storage but also secrets. Per now we have a setup with service principals accessing secrets through secret scopes, but we foresee a situation where we may get many service principals and the corresponding maintenance burden.

Looking at https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/azure-managed-identities it seems that Access Connectors would be a solution for the storage access part. But can we use "Access Connector for Azure Databricks" to access Azure Key Vault?

Re: Can we use "Access Connector for Azure Databricks" to access Azure Key Vault?

Hubert-Dudek — Tue, 06 Dec 2022 14:38:01 GMT

In what place exactly do you need to access key vault secrets?

Key vault can be integrated with databricks workspace under url

https://<YOUR_WORKSPACE>.azuredatabricks.net/#secrets/createScope

or via CLI/API

Re: Can we use "Access Connector for Azure Databricks" to access Azure Key Vault?

grazie — Tue, 06 Dec 2022 15:17:54 GMT

Thanks for your response 🙂

We need to access secrets from notebooks and other tasks running interactively or in workflows.

We're actually using Azure Key Vault-backed secret scopes now, but we rely on service principals to access the keyvault through secret scope. Secret scopes are problematic, e.g. because they can't be created in a fully automated way, and access control must be managed in Databricks Secret ACLs instead of using Key Vault access control (like Azure RBAC). Service principals come with a maintenance burden for IT who needs to rotate credentials at regular intervals.

We're looking for ways to avoid having to manage service principals, and use Managed Identities instead.

Re: Can we use "Access Connector for Azure Databricks" to access Azure Key Vault?

_paskal_ — Thu, 08 Dec 2022 15:08:42 GMT

Hi Grazie,

Did you manage to get this to work?

I am trying to do the same but no luck so far. I keep getting INVALID_STATE: Databricks could not access keyvault: https://xxxx.vault.azure.net/.

Although I openen all network and assigned all Key Vault related roles I keep getting this error so I am wondering if it is supported at all...

Re: Can we use "Access Connector for Azure Databricks" to access Azure Key Vault?

grive — Fri, 09 Dec 2022 13:43:34 GMT

I have unofficial word that this is not supported, and docs don't mention it. I have the feeling that even if I got it to work it should not be trusted for now.

Re: Can we use "Access Connector for Azure Databricks" to access Azure Key Vault?

_paskal_ — Mon, 12 Dec 2022 07:34:03 GMT

Thanks for your response, Grive.

I ended up using the default Service principal for Databricks (AzureDatabricks).