topic Re: parameterize azure storage account name in spark cluster config databricks in Data Engineering

parameterize azure storage account name in spark cluster config databricks

baatchus — Tue, 22 Feb 2022 11:36:24 GMT

wondering if this is to parameterize the azure storage account name part in the spark cluster config in Databricks?

I have a working example where the values are referencing secret scopes:

spark.hadoop.fs.azure.account.oauth2.client.id.<azurestorageaccountname>.dfs.core.windows.net {{secrets/keyvault_secret_scope/sp-id}}
spark.hadoop.fs.azure.account.oauth.provider.type.<azurestorageaccountname>.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
spark.hadoop.fs.azure.account.oauth2.client.endpoint.<azurestorageaccountname>.dfs.core.windows.net https://login.microsoftonline.com/05646f53-5e81-4a6a-85f5-a1ee2189bea4/oauth2/token
spark.hadoop.fs.azure.account.auth.type.<azurestorageaccountname>.dfs.core.windows.net OAuth
spark.hadoop.fs.azure.account.oauth2.client.secret.<azurestorageaccountname>.dfs.core.windows.net {{secrets/keyvault_secret_scope/sp-secret}}

But this does not work if I try it on the keys:

spark.hadoop.fs.azure.account.oauth2.client.id.{{secrets/keyvault_secret_scope/azurestorageaccountname}}.dfs.core.windows.net {{secrets/keyvault_secret_scope/sp-id}}
 
spark.hadoop.fs.azure.account.oauth.provider.type.{{secrets/keyvault_secret_scope/azurestorageaccountname}}.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
 
spark.hadoop.fs.azure.account.oauth2.client.endpoint.{{secrets/keyvault_secret_scope/azurestorageaccountname}}.dfs.core.windows.net https://login.microsoftonline.com/05646f53-5e81-4a6a-85f5-a1ee2189bea4/oauth2/token
 
spark.hadoop.fs.azure.account.auth.type.{{secrets/keyvault_secret_scope/azurestorageaccountname}}.dfs.core.windows.net OAuth
 
spark.hadoop.fs.azure.account.oauth2.client.secret.{{secrets/keyvault_secret_scope/azurestorageaccountname}}.dfs.core.windows.net {{secrets/keyvault_secret_scope/sp-secret}}

If it is impossible how can you replace the cluster config between environments?

I need the azure storage account name to be different in DEV vs PROD.

Re: parameterize azure storage account name in spark cluster config databricks

Anonymous — Tue, 22 Feb 2022 18:24:24 GMT

Howdy, @baatch us! Thank you for bringing your question to the community! We'll give other members a chance to respond before we come back around if we need to. 🙂

Re: parameterize azure storage account name in spark cluster config databricks

Atanu — Thu, 03 Mar 2022 14:41:20 GMT

@baatch us the cluster config works irrespective to different environment. so you can change the cluster config based on your requirement. So is it like you want to use different storage account ?

Re: parameterize azure storage account name in spark cluster config databricks

baatchus — Thu, 03 Mar 2022 22:29:39 GMT

I found the solution:

I could just remove the specific storage account and have the credentials be generic for all storage accounts

https://stackoverflow.com/questions/71220563/parameterize-azure-storage-account-name-in-spark-cluster-config-databricks/71258782#71258782

Re: parameterize azure storage account name in spark cluster config databricks

Anonymous — Sun, 06 Mar 2022 22:10:02 GMT

Fantastic! Thanks for letting us know!