Thank you for your response.

This is my understanding. Please correct me if I'm wrong. When Spark job is triggered (using spark-submit or Jar option), either from UI or using REST API, configured init scripts are executed first on the cluster node(s) before executing the Spark job. So in init scripts, we don't have an option to provide spark-submit command line arguments. We've dynamically populated environment variables in the configuration files that need to be sourced in from init scripts and made available to the Spark job. The sourcing in part is working fine, but these environment variables are not becoming available to the Java Spark Job.