parameterize azure storage account name in spark cluster config databricks

baatchus
New Contributor III

‎02-22-2022 03:36 AM

wondering if this is to parameterize the azure storage account name part in the spark cluster config in Databricks?

I have a working example where the values are referencing secret scopes:

spark.hadoop.fs.azure.account.oauth2.client.id.<azurestorageaccountname>.dfs.core.windows.net {{secrets/keyvault_secret_scope/sp-id}}
spark.hadoop.fs.azure.account.oauth.provider.type.<azurestorageaccountname>.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
spark.hadoop.fs.azure.account.oauth2.client.endpoint.<azurestorageaccountname>.dfs.core.windows.net https://login.microsoftonline.com/05646f53-5e81-4a6a-85f5-a1ee2189bea4/oauth2/token
spark.hadoop.fs.azure.account.auth.type.<azurestorageaccountname>.dfs.core.windows.net OAuth
spark.hadoop.fs.azure.account.oauth2.client.secret.<azurestorageaccountname>.dfs.core.windows.net {{secrets/keyvault_secret_scope/sp-secret}}

But this does not work if I try it on the keys:

spark.hadoop.fs.azure.account.oauth2.client.id.{{secrets/keyvault_secret_scope/azurestorageaccountname}}.dfs.core.windows.net {{secrets/keyvault_secret_scope/sp-id}}
 
spark.hadoop.fs.azure.account.oauth.provider.type.{{secrets/keyvault_secret_scope/azurestorageaccountname}}.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
 
spark.hadoop.fs.azure.account.oauth2.client.endpoint.{{secrets/keyvault_secret_scope/azurestorageaccountname}}.dfs.core.windows.net https://login.microsoftonline.com/05646f53-5e81-4a6a-85f5-a1ee2189bea4/oauth2/token
 
spark.hadoop.fs.azure.account.auth.type.{{secrets/keyvault_secret_scope/azurestorageaccountname}}.dfs.core.windows.net OAuth
 
spark.hadoop.fs.azure.account.oauth2.client.secret.{{secrets/keyvault_secret_scope/azurestorageaccountname}}.dfs.core.windows.net {{secrets/keyvault_secret_scope/sp-secret}}

If it is impossible how can you replace the cluster config between environments?

I need the azure storage account name to be different in DEV vs PROD.

0 Kudos

Anonymous
Not applicable

‎02-22-2022 10:24 AM

Howdy, @baatch us​! Thank you for bringing your question to the community! We'll give other members a chance to respond before we come back around if we need to. 🙂

0 Kudos

Atanu
Databricks Employee
Databricks Employee

‎03-03-2022 06:41 AM

@baatch us​  the cluster config works irrespective to different environment. so you can change the cluster config based on your requirement. So is it like you want to use different storage account ?

0 Kudos

baatchus
New Contributor III

‎03-03-2022 02:29 PM

I found the solution:

I could just remove the specific storage account and have the credentials be generic for all storage accounts

https://stackoverflow.com/questions/71220563/parameterize-azure-storage-account-name-in-spark-cluste...

View solution in original post

0 Kudos

Anonymous
Not applicable

‎03-06-2022 02:10 PM

Fantastic! Thanks for letting us know!

0 Kudos