if your application code primarily consists of Python files and does not require a separate virtual environment with specific dependencies, you can use the --py-files argument in spark-submit

spark-submit --verbose \
   --master yarn \
  --deploy-mode cluster \
  --name $APPNAME \
  --driver-memory 1g \  # Adjust memory as needed
  --executor-memory 1g \  # Adjust memory as needed
  --num-executors 2 \     # Adjust executors as needed
  --py-files ${build_directory}/source_code.zip \
  $CODE_DIRECTORY_CLOUD/my_application_entry_point.py  # Path to your main application script

For application code with a separate virtual environment)

If your application code has specific dependencies that you manage in a separate virtual environment, you can leverage the --conf spark.yarn.dist.archives argument.

Explanation:

• --conf "spark.yarn.dist.archives"=${pyspark_venv}.tar.gz#pyspark_venv: This configures Spark to distribute your virtual environment archive (pyspark_venv.tar.gz) to the Yarn cluster nodes. The #pyspark_venv  part defines a symbolic link name within the container.
• You do not need --py-fileshere because the virtual environment archive will contain all necessary dependencies.

Choosing the best approach:

The choice depends on your project setup:

• No Separate Virtual Environment: Use  --py-files if your application code consists mainly of Python files and doesn't require a separate virtual environment.
• Separate Virtual Environment: Use --conf spark.yarn.dist.archives if you manage dependencies in a separate virtual environment archive.

HTH

Mich Talebzadeh,

Dad | Technologist | Solutions Architect | Engineer

London

United Kingdom