How to change the number of executors instances in databricks

[Harun]

I know that Databricks runs one executor per worker node.

Can i change the no.of.exectors by adding params (spark.executor.instances) in the cluster advance option? and also can i pass this parameter when i schedule a task, so that particular task will run with that configuration?

Is it advisable to modify the executor params like the no.of.instances and memory of executors or we need to go with the default one which is taken care by databricks internally.

[karthik_p]

@Harun Raseed Basheer​ usually for 1 worker node 1 executor will be there, if we need to split that executor within that worker node itself, we can do that based on memory that core has been assigned , below configs can be use

1. spark.executor.cores
2. spark.executor.memory

below article will help you

How to configure single-core executors to run JNI libraries - Databricks