Hi there,
I am trying to set up databricks storage account access in Global init script. according to Azure Databricks document on creating cluster with driver config for all clusters (https://learn.microsoft.com/en-us/azure/databricks/archive/compute/configure#spark-configuration), I wrote following code, but I am not sure the format is correct or not and how to reference secret.
I tested the init script, got this error:
ERROR ProjectConf$: Failed to parse conf file '/databricks/driver/conf/00-custom-spark-driver-defaults.conf', skipping...
com.typesafe.config.ConfigException$Parse: File: /databricks/driver/conf/00-custom-spark-driver-defaults.conf: 8: in value for key '"fs.azure.account.oauth2.client.secret.adlssaprdraw.dfs.core.windows.net"': expecting a close brace or a field name here, got '{'
at com.typesafe.config.impl.Parser$ParseContext.parseError(Parser.java:435)
could you help?
Thanks!
My code:
spark_defaults_conf="/databricks/driver/conf/00-custom-spark-driver-defaults.conf"
cat << EOF > $spark_defaults_conf
[driver] {
"fs.azure.account.auth.type.adlssadevraw.dfs.core.windows.net" = "OAuth"
"fs.azure.account.oauth.provider.type.adlssadevraw.dfs.core.windows.net" = "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"
"fs.azure.account.oauth2.client.id.adlssadevraw.dfs.core.windows.net" = "444aef64-8f39-41c0-b769-e312d20be27f"
"fs.azure.account.oauth2.client.secret.adlssadevraw.dfs.core.windows.net" = {{secrets/dev-kv-01-scope/databricks-dev-01-sp}}
"fs.azure.account.oauth2.client.endpoint.adlssadevraw.dfs.core.windows.net" = "https://login.microsoftonline.com/c499ec336-2375-432e-92f5-63cbbc442ad57/oauth2/token"
}
EOF
