Hi everyone,
Hoping someone can help me with this problem. I have an embarrassingly parallel workload, which I'm parallelising over 4 worker nodes (of type Standard_F4, so 4 cores each). Each workload is single-threaded, so I believe that only one core is actually being utilised for each task. I'd like to ideally run 2+ tasks on each worker.
I've tried increasing the number of executors (having more than one per worker) by means of the following, but it doesn't seem to work.
spark.executor.cores 1
spark.executor.memory 2g
spark.executor.instances 16 // this is 4 workers * 4 cores = 16 executors
I've also tried dynamic allocation of executors, per the answer to this Stack Overflow thread, but that's also not working: java - How to set amount of Spark executors? - Stack Overflow.
Any help would be much appreciated. I can furnish more details if required.