Hi all thank you for the suggestions. 

Doing This 

spark.conf.set("fs.azure.account.key.{storage_account}.dfs.core.windows.net", "{myStorageAccountKey}")

For the hadoop configuration does not work.

And the suggestion of @Tharun-Kumar would suggest to hardcode secrets in the configuration which is a big no.

Someone else suggested the proper solution on stack overflow which is to add in the same location @Tharun-Kumar suggested to add this, but pointing at the secret scope at the same time:

spark.hadoop.fs.azure.account.key.<account_name>.blob.core.windows.net {{secrets/<secret-scope-name>/<secret-name>}}