I have a big cluster with the auto-scaling(min:1, max: 25) feature enabled. I want to run multiple jobs on that cluster with different values of spark properties( `--executor-cores` and `–executor-memory) but I don't see any option to specify the same when creating the jobs.

I tried the below code snippet in my Pyspark application.

spark = SparkSession \
    .builder \
    .config("spark.executor.instances", "2") \
    .appName("SparkWarehouseETL") \
    .getOrCreate()

But when running this application, it used all the 25 workers instead of using only 2. Since it is already using all the 25 workers, submitting another is just a waste.

What am I missing? Please help !!!