How to change the number of executors instances in databricks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-29-2022 01:28 PM
I know that Databricks runs one executor per worker node.
Can i change the no.of.exectors by adding params (spark.executor.instances) in the cluster advance option? and also can i pass this parameter when i schedule a task, so that particular task will run with that configuration?
Is it advisable to modify the executor params like the no.of.instances and memory of executors or we need to go with the default one which is taken care by databricks internally.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-29-2022 06:09 PM
@Harun Raseed Basheer usually for 1 worker node 1 executor will be there, if we need to split that executor within that worker node itself, we can do that based on memory that core has been assigned , below configs can be use
- spark.executor.cores
- spark.executor.memory
below article will help you
How to configure single-core executors to run JNI libraries - Databricks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-29-2025 11:35 AM - edited 12-29-2025 11:38 AM
In Databricks, the executor model is largely managed by the platform itself. On Databricks clusters, each worker node typically runs a single Spark executor, and this behavior is intentional.
