https://community.databricks.com/t5/data-engineering/how-to-restrict-the-number-of-tasks-per-executor/m-p/14770#M9215 <P>We can use a config called "spark.task.cpus"</P><P>This specifies the number of cores to allocate for each task.</P><P></P><P>The default value is 1</P><P>If we specify say 2, it means fewer tasks will be assigned to the executor.</P> Wed, 22 Sep 2021 22:23:12 GMT saipujari_spark 2021-09-22T22:23:12Z https://community.databricks.com/t5/data-engineering/how-to-restrict-the-number-of-tasks-per-executor/m-p/14768#M9213 <P>In general, one task per core is how spark executes the tasks.</P><P>If we want to restrict the number of tasks submitted to the executor to get more task to memory ratio, How can we achieve that?</P> Wed, 22 Sep 2021 22:17:47 GMT https://community.databricks.com/t5/data-engineering/how-to-restrict-the-number-of-tasks-per-executor/m-p/14768#M9213 saipujari_spark 2021-09-22T22:17:47Z https://community.databricks.com/t5/data-engineering/how-to-restrict-the-number-of-tasks-per-executor/m-p/14770#M9215 <P>We can use a config called "spark.task.cpus"</P><P>This specifies the number of cores to allocate for each task.</P><P></P><P>The default value is 1</P><P>If we specify say 2, it means fewer tasks will be assigned to the executor.</P> Wed, 22 Sep 2021 22:23:12 GMT https://community.databricks.com/t5/data-engineering/how-to-restrict-the-number-of-tasks-per-executor/m-p/14770#M9215 saipujari_spark 2021-09-22T22:23:12Z