If --py-files doesn’t work, try this shorter method:

1. Create a Conda Environment: Install your packages.

   conda create -n myenv python=3.x
   conda activate myenv
   pip install your-package

   Package and Submit: Use conda-pack and spark-submit with --archives.

   conda pack -n myenv -o myenv.tar.gz
   spark-submit --archives myenv.tar.gz#myenv --conf spark.pyspark.python=myenv/bin/python your_script.py
   
   

   This runs your Spark job with the required packages.

   Regards,
   Summa Marketing