topic Re: What's the recommended way to use secrets within Global init scripts? in Data Engineering

What's the recommended way to use secrets within Global init scripts?

Anonymous — Fri, 25 Jun 2021 05:19:28 GMT

Re: What's the recommended way to use secrets within Global init scripts?

User16826994223 — Fri, 25 Jun 2021 09:54:02 GMT

The order of execution of init scripts is:

Legacy global (deprecated)
Cluster-named (deprecated)
Global (new)
Cluster-scoped

Arfrenece link that can help is - https://docs.databricks.com/clusters/init-scripts.html#global-init-scripts

Re: What's the recommended way to use secrets within Global init scripts?

sajith_appukutt — Fri, 25 Jun 2021 17:32:27 GMT

You could specify a secret path in an environment at the cluster level

SPARKPASSWORD=<path-value>

and this would be available in global init scripts / cluster-scoped init script

if [[ $SPARKPASSWORD ]]; then
  use $SPARKPASSWORD
fi

Re: What's the recommended way to use secrets within Global init scripts?

sstrong99 — Thu, 08 Dec 2022 15:57:35 GMT

I also have this question, and I don't understand the answer from @Sajith Appukuttan . This solution would require configuring each new cluster separately with the correct environment variable, correct? That seems like it defeats the purpose of using a global init script. Is there a method that does not require configuration of each new cluster separately?

Re: What's the recommended way to use secrets within Global init scripts?

peta — Wed, 04 Dec 2024 15:23:25 GMT

Hello everyone,

How did you solve this issue? Did you manage to get the secrets through global init script?

Thanks.