topic Re: Referring to Azure Keyvault secrets in spark config in Data Engineering

Referring to Azure Keyvault secrets in spark config

Martin1 — Thu, 05 May 2022 11:11:12 GMT

Hi all

In spark config for a cluster, it works well to refer to a Azure Keyvault secret in the "value" part of the name/value combo on a config row/setting.

For example, this works fine (I've removed the string that is our specific storage account name):

fs.azure.account.oauth2.client.secret.<storage_account_name>.dfs.core.windows.net {{secrets/secret_scope/secret_value}}

But is it possible to refer to a secret inside of the name-string of the config-row? More specifically, in the example above I would like to have the <storage_account_name> dynamic, using a secret (or any other way) so that it does not need to be hard-coded. Then we would have a more generic and re-usable spark config.

I actually tried this, but it doesn't seem to work:

fs.azure.account.oauth2.client.secret.{{secrets/secret_scope/storage_account_name_secret}}.dfs.core.windows.net {{secrets/secret_scope/secret_value}}

Is there a way to achieve this?

Many thanks,

Martin

Re: Referring to Azure Keyvault secrets in spark config

jose_gonzalez — Fri, 29 Jul 2022 00:24:23 GMT

Hi @Martin Aronsson,

Just a friendly follow-up. Do you still are looking for help or Kaniz's response help you to resolved your issue?

Re: Referring to Azure Keyvault secrets in spark config

kp12 — Mon, 02 Oct 2023 10:31:23 GMT

Hello,

Is there any update on this issue please?

Databricks no longer recommend mounting external location, so the other way to access Azure storage is to use spark config as mentioned in this document - https://learn.microsoft.com/en-us/azure/databricks/storage/azure-storage#connect-to-azure-data-lake-storage-gen2-or-blob-storage-using-azure-credentials

Although the spark config works fine, but as @Martin1 mentioned, the value for storage-account and directory-id cannot be accessed from the secret because they are part of the property name.

Is there a way of accessing these from Databricks secret? as hardcoding these values doesn't seem right.

Thanks,

Kalyani