Anonymous
Not applicable
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-03-2023 03:16 AM
To make environment variables defined in an init script available to a Spark JVM job, you can pass the environment variables to the Spark job as command line arguments or system properties.
Here's an example of passing environment variables as command line arguments:
spark-submit --master yarn --deploy-mode client \
--conf spark.driver.extraJavaOptions=-Denv_var_1=$env_var_1 \
--conf spark.executor.extraJavaOptions=-Denv_var_2=$env_var_2 \
/path/to/your/SparkJob.jar