I found a workaround for the problem to be able to use the secrets from the KeyVault in all the execturos. I only tested so far this in the notebooks, I want to try later in a JAR job.

First here is a link to the official documentation that highlights some limitations of dbutils API, the very one that Databricks recommends to read secrets.

Now the workaround:

Not being able to use dbutils from execturos, these are the steps I followed to bring the values of a KeyVault secret to an object initialization in every executor:

1. Reference a secret in an evnironmental variable, like this.

2. Prior to any other cell execution, set values in the same variable names from the driver, but in the executor´s environments, like this:

   spark.conf.set("spark.executorEnv.anon_pwd", sys.env("anon_pwd")) spark.conf.set("spark.executorEnv.anon_salt", sys.env("anon_salt")) spark.conf.set("spark.executorEnv.anon_tweak", sys.env("anon_tweak"))

3. Use the UDF function that makes use of the singleton object in every executor.