Hello,

After implementing the use of Secret Scope to store Secrets in an azure key vault, i faced a problem.

When writting an output to the blob i get the following error:

shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access container analysis in account [REDACTED].blob.core.windows.net using anonymous credentials, and no credentials found for them in the configuration.

After some investigation, it is related to the following config previously set in the advanced configuration of the cluster configuration:

"spark.hadoop.fs.azure.account.key.y.blob.core.windows.net", "myStorageAccountKey"

I would like to find the way to set this in the notebook level after retrieving the secret from the secret scope:

spark.conf.set("spark.hadoop.fs.azure.account.key.y.blob.core.windows.net", "myStorageAccountKey")

Unfortunatly this does not work.

Here below an example of how i write the output:

df.write.format("com.crealytics.spark.excel") \
  .option("dataAddress", "'%s'!A1" %(sheetName)) \
  .option("header", "true") \
  .option("dateFormat", "yy-mm-d") \
  .option("timestampFormat", "mm-dd-yyyy hh:mm:ss") \
  .option("useHeader", "true") \
  .mode("append")  \
  .save( "%s/%s" %(output_blob_folder,outputName))