Databricks cannot access Azure Key Vault

[SimonNuss]

I am trying to set retrieve a secret from Azure Key Vault as follows:

sqlPassword = dbutils.secrets.get(scope = "Admin", key = "SqlPassword")

The scope has been created correctly, but I receive the following error message:

com.databricks.common.client.DatabricksServiceException: INVALID_STATE: Databricks could not access the keyvault: https://AzureKeyVaultName.vault.azure.net/.

I assume I need to add Databricks in Azure Key Vaults "Access Policies" however I cannot find any information online on how to do this. Any ideas?

[HariBaskar]

This might help - https://docs.azuredatabricks.net/user-guide/secrets/example-secret-workflow.html

[Prosenjit]

You can follow the steps mentioned at:

https://medium.com/@cprosenjit/azure-databricks-with-azure-key-vaults-c00df6548222

[Rodneyjoyce]

This did not work for me. I have the same error - including misspelling -

"Internal error happened while granting read/list permission to Databricks ervice principal to KeyVault: XYZ"

I'm assuming Databricks is using a default service principal in Azure AD to communicate with KeyVault but I don't have access to AD and I can't find the Databricks principal name.

[sagarsharmas]

Hey did you solve the issue?

[Rodneyjoyce · Accepted solution]

FYI I found the answer to my problem and posted about it here on SO to share: https://stackoverflow.com/questions/56537214/creating-a-secret-scope-in-databricks-backed-by-azure-k...

It was because I was using a user in Databricks that did not have rights in Azure AD to make a service principal.

[virahkumar]

Sometimes turning it off and on again is underrated, so I gave up finding the problem, deleted it and re-created the scope - worked a breeze!

Mine seems like it was something silly, I was able to set up my vault but got the same issue when trying to use it 1hr later - even when logged in as myself, an admin of the workspace. Whenever I created it before with the exact same process (via a pipeline), it worked, just not this time... so no problem ID here, just a solution 🙂