Hi there,

I am trying to set up databricks storage account access in Global init script. according to Azure Databricks document on creating cluster with driver config for all clusters (https://learn.microsoft.com/en-us/azure/databricks/archive/compute/configure#spark-configuration), I wrote following code, but I am not sure the format is correct or not and how to reference secret.

I tested the init script, got this error:

ERROR ProjectConf$: Failed to parse conf file '/databricks/driver/conf/00-custom-spark-driver-defaults.conf', skipping...

com.typesafe.config.ConfigException$Parse: File: /databricks/driver/conf/00-custom-spark-driver-defaults.conf: 8: in value for key '"fs.azure.account.oauth2.client.secret.adlssaprdraw.dfs.core.windows.net"': expecting a close brace or a field name here, got '{'

   at com.typesafe.config.impl.Parser$ParseContext.parseError(Parser.java:435)

could you help?

Thanks!

My code:

spark_defaults_conf="/databricks/driver/conf/00-custom-spark-driver-defaults.conf"

cat << EOF > $spark_defaults_conf

[driver] {

"fs.azure.account.auth.type.adlssadevraw.dfs.core.windows.net" = "OAuth"

"fs.azure.account.oauth.provider.type.adlssadevraw.dfs.core.windows.net" = "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider"

"fs.azure.account.oauth2.client.id.adlssadevraw.dfs.core.windows.net" = "444aef64-8f39-41c0-b769-e312d20be27f"

"fs.azure.account.oauth2.client.secret.adlssadevraw.dfs.core.windows.net" = {{secrets/dev-kv-01-scope/databricks-dev-01-sp}}

"fs.azure.account.oauth2.client.endpoint.adlssadevraw.dfs.core.windows.net" = "https://login.microsoftonline.com/c499ec336-2375-432e-92f5-63cbbc442ad57/oauth2/token"

}

EOF