cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Referring to Azure Keyvault secrets in spark config

Martin1
New Contributor II

Hi all

In spark config for a cluster, it works well to refer to a Azure Keyvault secret in the "value" part of the name/value combo on a config row/setting.

For example, this works fine (I've removed the string that is our specific storage account name):

fs.azure.account.oauth2.client.secret.<storage_account_name>.dfs.core.windows.net {{secrets/secret_scope/secret_value}}

But is it possible to refer to a secret inside of the name-string of the config-row? More specifically, in the example above I would like to have the <storage_account_name> dynamic, using a secret (or any other way) so that it does not need to be hard-coded. Then we would have a more generic and re-usable spark config.

I actually tried this, but it doesn't seem to work:

fs.azure.account.oauth2.client.secret.{{secrets/secret_scope/storage_account_name_secret}}.dfs.core.windows.net {{secrets/secret_scope/secret_value}}

Is there a way to achieve this?

Many thanks,

Martin

3 REPLIES 3

Kaniz_Fatma
Community Manager
Community Manager

Hi @Martin Aronsson​, This is a link to secret management in Databricks.

Hi @Martin Aronsson​,

Just a friendly follow-up. Do you still are looking for help or Kaniz's response help you to resolved your issue?

kp12
New Contributor II

Hello,

Is there any update on this issue please? 

Databricks no longer recommend mounting external location, so the other way to access Azure storage is to use spark config as mentioned in this document - https://learn.microsoft.com/en-us/azure/databricks/storage/azure-storage#connect-to-azure-data-lake-...

Although the spark config works fine, but as @Martin1 mentioned, the value for storage-account and directory-id cannot be accessed from the secret because they are part of the property name. 

Is there a way of accessing these from Databricks secret? as hardcoding these values doesn't seem right.

Thanks,

Kalyani

 

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group