โ07-24-2023 04:47 AM
Hi everyone,
Hoping someone can help me with this problem. I have an embarrassingly parallel workload, which I'm parallelising over 4 worker nodes (of type Standard_F4, so 4 cores each). Each workload is single-threaded, so I believe that only one core is actually being utilised for each task. I'd like to ideally run 2+ tasks on each worker.
I've tried increasing the number of executors (having more than one per worker) by means of the following, but it doesn't seem to work.
spark.executor.cores 1
spark.executor.memory 2g
spark.executor.instances 16 // this is 4 workers * 4 cores = 16 executors
I've also tried dynamic allocation of executors, per the answer to this Stack Overflow thread, but that's also not working: java - How to set amount of Spark executors? - Stack Overflow.
Any help would be much appreciated. I can furnish more details if required.
โ07-26-2023 05:51 AM
So I managed to get the 1-core-per-executor working successfully. The bit that wasn't working was spark.executor.memory -- this was too high, but lowering it so that the sum of the executors memory was ~90% of the worker node's memory allowed it to work properly.
โ07-24-2023 10:11 PM
Hi @DennisB
Are you using threadpool to run workloads in parallel?
Also, I would suggest removing the three configs, you had mentioned. Now, if you create a 16 pool thread, then you would be seeing 16 runs happening on the executors front (16 cores being utilized).
โ07-26-2023 05:53 AM
I haven't tried threadpool, but thanks for the suggestion. Before posting I had tried a multiprocessing Pool, but that didn't work (I'd hoped to use multiprocessing on each worker now, i.e., Spark to distribute to worker nodes, then multiprocessing to distribute to each core, but I couldn't get it to work -- I didn't think to try threadpool though).
โ07-24-2023 10:59 PM
Hi @DennisB
We haven't heard from you since the last response from @Tharun-Kumar โ, and I was checking back to see if her suggestions helped you.
Or else, If you have any solution, please share it with the community, as it can be helpful to others.
Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.
โ07-26-2023 05:51 AM
So I managed to get the 1-core-per-executor working successfully. The bit that wasn't working was spark.executor.memory -- this was too high, but lowering it so that the sum of the executors memory was ~90% of the worker node's memory allowed it to work properly.
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group