topic Re: Better Worker Node Core Utilisation in Data Engineering

Better Worker Node Core Utilisation

DennisB — Mon, 24 Jul 2023 11:47:42 GMT

Hi everyone,

Hoping someone can help me with this problem. I have an embarrassingly parallel workload, which I'm parallelising over 4 worker nodes (of type Standard_F4, so 4 cores each). Each workload is single-threaded, so I believe that only one core is actually being utilised for each task. I'd like to ideally run 2+ tasks on each worker.

I've tried increasing the number of executors (having more than one per worker) by means of the following, but it doesn't seem to work.

spark.executor.cores 1
spark.executor.memory 2g
spark.executor.instances 16 // this is 4 workers * 4 cores = 16 executors

I've also tried dynamic allocation of executors, per the answer to this Stack Overflow thread, but that's also not working: java - How to set amount of Spark executors? - Stack Overflow.

Any help would be much appreciated. I can furnish more details if required.

Re: Better Worker Node Core Utilisation

Tharun-Kumar — Tue, 25 Jul 2023 05:11:15 GMT

Hi @DennisB

Are you using threadpool to run workloads in parallel?

Also, I would suggest removing the three configs, you had mentioned. Now, if you create a 16 pool thread, then you would be seeing 16 runs happening on the executors front (16 cores being utilized).

Re: Better Worker Node Core Utilisation

Anonymous — Tue, 25 Jul 2023 05:59:31 GMT

Hi @DennisB

We haven't heard from you since the last response from @Tharun-Kumar , and I was checking back to see if her suggestions helped you.

Or else, If you have any solution, please share it with the community, as it can be helpful to others.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

Re: Better Worker Node Core Utilisation

DennisB — Wed, 26 Jul 2023 12:51:24 GMT

So I managed to get the 1-core-per-executor working successfully. The bit that wasn't working was spark.executor.memory -- this was too high, but lowering it so that the sum of the executors memory was ~90% of the worker node's memory allowed it to work properly.

Re: Better Worker Node Core Utilisation

DennisB — Wed, 26 Jul 2023 12:53:40 GMT

I haven't tried threadpool, but thanks for the suggestion. Before posting I had tried a multiprocessing Pool, but that didn't work (I'd hoped to use multiprocessing on each worker now, i.e., Spark to distribute to worker nodes, then multiprocessing to distribute to each core, but I couldn't get it to work -- I didn't think to try threadpool though).