Pass secret in spark config when value is in form a.b.c={{secrets/scope/secret}}

[macmiller1]

I am configuring the Cluster for a spark-submit task and I am trying to specify `spark.executor.extraJavaOptions a.b.c={{secrets/scope/secret}}` but the literal {{secrets/scope/secret}} is being passed in rather than the secret value itself.

I know the convention for secrets is `spark.conf.something {{secrets/scope/secret}}` when the value is the secret itself and this substitution works fine for me in similar cases. Given my value in the above is a.b.c={{secrets/scope/secret}} the secret substitution is not working. I have also tried with an environment variable ie `spark.executor.extraJavaOptions a.b.c=${MY_ENV_VAR}` but this is also not working. 

I suspect that given the {{secrets/scope/secret}} does not have a space character before it but instead `=` the substitution isn't working properly.

Any suggestions on how to achieve this would be greatly appreciated!

[Kaniz_Fatma · Accepted solution]

Hi @macmiller1, 

• Instead of using the {{secrets/scope/secret}} syntax, you can try using environment variables.
• If you prefer to use the {{secrets/scope/secret}} syntax, you can try escaping the equal sign (=) in your value.
• One way to do this is by using the URL-encoded version of the equal sign, which is %3D.
• Instead of using an equal sign (=) in your value, you can try using a different separator, such as a colon (:) or a comma (,).

Try these solutions and see if they help resolve the issue you're facing with the secret substitution in the spark.executor.extraJavaOptions parameter.