cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to change the number of executors instances in databricks

Harun
Honored Contributor

I know that Databricks runs one executor per worker node.

Can i change the no.of.exectors by adding params (spark.executor.instances) in the cluster advance option? and also can i pass this parameter when i schedule a task, so that particular task will run with that configuration?

Is it advisable to modify the executor params like the no.of.instances and memory of executors or we need to go with the default one which is taken care by databricks internally.

1 REPLY 1

karthik_p
Esteemed Contributor

@Harun Raseed Basheerโ€‹ usually for 1 worker node 1 executor will be there, if we need to split that executor within that worker node itself, we can do that based on memory that core has been assigned , below configs can be use

  1. spark.executor.cores
  2. spark.executor.memory

below article will help you

How to configure single-core executors to run JNI libraries - Databricks

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group