How to restrict the number of tasks per executor?

[saipujari_spark]

In general, one task per core is how spark executes the tasks.

If we want to restrict the number of tasks submitted to the executor to get more task to memory ratio, How can we achieve that?

[saipujari_spark · Accepted solution]

We can use a config called "spark.task.cpus"

This specifies the number of cores to allocate for each task.

The default value is 1

If we specify say 2, it means fewer tasks will be assigned to the executor.