How to restrict the number of tasks per executor?

[saipujari_spark]

In general, one task per core is how spark executes the tasks.

If we want to restrict the number of tasks submitted to the executor to get more task to memory ratio, How can we achieve that?

[Kaniz]

Hi @ Sai! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers on the community have an answer to your question first. Or else I will follow up with my team and get back to you soon.Thanks.

[saipujari_spark · Accepted solution]

We can use a config called "spark.task.cpus"

This specifies the number of cores to allocate for each task.

The default value is 1

If we specify say 2, it means fewer tasks will be assigned to the executor.