Azure Databricks cluster driver config

ShellyXiao — Mon, 24 Apr 2023 19:50:29 GMT

Hi there,

I am trying to set up databricks storage account access in Global init script. according to Azure Databricks document on creating cluster with driver config for all clusters (https://learn.microsoft.com/en-us/azure/databricks/archive/compute/configure#spark-configuration), I wrote following code, but I am not sure the format is correct or not and how to reference secret.

I tested the init script, got this error:

ERROR ProjectConf$: Failed to parse conf file '/databricks/driver/conf/00-custom-spark-driver-defaults.conf', skipping...

com.typesafe.config.ConfigException$Parse: File: /databricks/driver/conf/00-custom-spark-driver-defaults.conf: 8: in value for key '"fs.azure.account.oauth2.client.secret.adlssaprdraw.dfs.core.windows.net"': expecting a close brace or a field name here, got '{'

at com.typesafe.config.impl.Parser$ParseContext.parseError(Parser.java:435)

could you help?

Thanks!

My code:

spark_defaults_conf="/databricks/driver/conf/00-custom-spark-driver-defaults.conf"

cat << EOF > $spark_defaults_conf

[driver] {