parameterize azure storage account name in spark cluster config databricks

[baatchus]

wondering if this is to parameterize the azure storage account name part in the spark cluster config in Databricks?

I have a working example where the values are referencing secret scopes:

spark.hadoop.fs.azure.account.oauth2.client.id.<azurestorageaccountname>.dfs.core.windows.net {{secrets/keyvault_secret_scope/sp-id}}
spark.hadoop.fs.azure.account.oauth.provider.type.<azurestorageaccountname>.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
spark.hadoop.fs.azure.account.oauth2.client.endpoint.<azurestorageaccountname>.dfs.core.windows.net https://login.microsoftonline.com/05646f53-5e81-4a6a-85f5-a1ee2189bea4/oauth2/token
spark.hadoop.fs.azure.account.auth.type.<azurestorageaccountname>.dfs.core.windows.net OAuth
spark.hadoop.fs.azure.account.oauth2.client.secret.<azurestorageaccountname>.dfs.core.windows.net {{secrets/keyvault_secret_scope/sp-secret}}

But this does not work if I try it on the keys:

spark.hadoop.fs.azure.account.oauth2.client.id.{{secrets/keyvault_secret_scope/azurestorageaccountname}}.dfs.core.windows.net {{secrets/keyvault_secret_scope/sp-id}}
 
spark.hadoop.fs.azure.account.oauth.provider.type.{{secrets/keyvault_secret_scope/azurestorageaccountname}}.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
 
spark.hadoop.fs.azure.account.oauth2.client.endpoint.{{secrets/keyvault_secret_scope/azurestorageaccountname}}.dfs.core.windows.net https://login.microsoftonline.com/05646f53-5e81-4a6a-85f5-a1ee2189bea4/oauth2/token
 
spark.hadoop.fs.azure.account.auth.type.{{secrets/keyvault_secret_scope/azurestorageaccountname}}.dfs.core.windows.net OAuth
 
spark.hadoop.fs.azure.account.oauth2.client.secret.{{secrets/keyvault_secret_scope/azurestorageaccountname}}.dfs.core.windows.net {{secrets/keyvault_secret_scope/sp-secret}}

If it is impossible how can you replace the cluster config between environments?

I need the azure storage account name to be different in DEV vs PROD.

[Anonymous]

Howdy, @baatch us​! Thank you for bringing your question to the community! We'll give other members a chance to respond before we come back around if we need to. 🙂

[Atanu]

@baatch us​  the cluster config works irrespective to different environment. so you can change the cluster config based on your requirement. So is it like you want to use different storage account ?

[baatchus · Accepted solution]

I found the solution:

I could just remove the specific storage account and have the credentials be generic for all storage accounts

https://stackoverflow.com/questions/71220563/parameterize-azure-storage-account-name-in-spark-cluste...

[Anonymous]

Fantastic! Thanks for letting us know!