Hi,

We're using Databricks Runtime version 11.3LTS and executing a Spark Java Job using a Job Cluster. To automate the execution of this job, we need to define (source in from bash config files) some environment variables through an init script (cluster-scoped) and make them available to the Spark Java job.

The defined or sourced in environment variables get set in the init script, however they don't become available to the Spark Java job. We tried updating the /etc/environment file as stated at the following location, but that didn't help us -

https://community.databricks.com/s/topic/0TO3f000000Ciy8GAC/variables

Please let us know if we're missing anything here. Any help to get this addressed will be highly appreciated.

Thanks in advance!

Regards,

// Rahul