topic Re: Databricks cannot access Azure Key Vault in Data Engineering

Databricks cannot access Azure Key Vault

SimonNuss — Wed, 10 Oct 2018 18:52:38 GMT

I am trying to set retrieve a secret from Azure Key Vault as follows:

sqlPassword = dbutils.secrets.get(scope = "Admin", key = "SqlPassword")

The scope has been created correctly, but I receive the following error message:

com.databricks.common.client.DatabricksServiceException: INVALID_STATE: Databricks could not access the keyvault: https://AzureKeyVaultName.vault.azure.net/.

I assume I need to add Databricks in Azure Key Vaults "Access Policies" however I cannot find any information online on how to do this. Any ideas?

Re: Databricks cannot access Azure Key Vault

HariBaskar — Wed, 09 Jan 2019 12:33:21 GMT

This might help - https://docs.azuredatabricks.net/user-guide/secrets/example-secret-workflow.html

Re: Databricks cannot access Azure Key Vault

Prosenjit — Sun, 20 Jan 2019 05:23:20 GMT

You can follow the steps mentioned at:

https://medium.com/@cprosenjit/azure-databricks-with-azure-key-vaults-c00df6548222

Re: Databricks cannot access Azure Key Vault

sagarsharmas — Tue, 05 Mar 2019 21:27:23 GMT

Hey did you solve the issue?

Re: Databricks cannot access Azure Key Vault

Rodneyjoyce — Tue, 11 Jun 2019 03:44:24 GMT

This did not work for me. I have the same error - including misspelling -

"Internal error happened while granting read/list permission to Databricks ervice principal to KeyVault: XYZ"

I'm assuming Databricks is using a default service principal in Azure AD to communicate with KeyVault but I don't have access to AD and I can't find the Databricks principal name.

Re: Databricks cannot access Azure Key Vault

Rodneyjoyce — Tue, 11 Jun 2019 05:48:15 GMT

FYI I found the answer to my problem and posted about it here on SO to share: https://stackoverflow.com/questions/56537214/creating-a-secret-scope-in-databricks-backed-by-azure-key-vault-fails/56537215#56537215

It was because I was using a user in Databricks that did not have rights in Azure AD to make a service principal.

Re: Databricks cannot access Azure Key Vault

virahkumar — Tue, 11 Aug 2020 04:15:38 GMT

Sometimes turning it off and on again is underrated, so I gave up finding the problem, deleted it and re-created the scope - worked a breeze!

Mine seems like it was something silly, I was able to set up my vault but got the same issue when trying to use it 1hr later - even when logged in as myself, an admin of the workspace. Whenever I created it before with the exact same process (via a pipeline), it worked, just not this time... so no problem ID here, just a solution 🙂