topic Re: Questions About Workers and Executors Configuration in Databricks in Data Engineering

Questions About Workers and Executors Configuration in Databricks

dvd_lg_bricks — Tue, 09 Dec 2025 07:10:06 GMT

Hi everyone, sorry, I’m new here. I’m considering migrating to Databricks, but I need to clarify a few things first.

When I define and launch an application, I see that I can specify the number of workers, and then later configure the number of executors.
My questions are:

Are those workers running on different machines?
Can I define how many executors run on each worker?
If so, is this controlled through Spark configuration variables?

Thanks!

Re: Questions About Workers and Executors Configuration in Databricks

szymon_dybczak — Tue, 09 Dec 2025 10:00:41 GMT

Hi @dvd_lg_bricks ,

1. Yes.In Databricks:

A cluster = driver node + one or more worker nodes
Each worker is a separate (VM)

2. Databricks runs one executor per worker node. Therefore, the terms executor and worker are used interchangeably in the context of the Databricks architecture.

Re: Questions About Workers and Executors Configuration in Databricks

Raman_Unifeye — Tue, 09 Dec 2025 10:56:55 GMT

@dvd_lg_bricks - Databricks does the spark-cluster management.

Are those workers running on different machines? - Yes, it runs one VM per worker
Can I define how many executors run on each worker? - No, Databricks runs 1 executor per worker
If so, is this controlled through Spark configuration variables? - not on databricks. but on vanilla spark, you would manage it using below 3 variables -
- spark.executor.instances
- spark.executor.cores
- spark.executor.memory

Re: Questions About Workers and Executors Configuration in Databricks

dvd_lg_bricks — Tue, 09 Dec 2025 11:16:22 GMT

Thank you both, @szymon_dybczak @Raman_Unifeye for your feedback. I really appreciate the time and clarity you provided.

Re: Questions About Workers and Executors Configuration in Databricks

szymon_dybczak — Tue, 09 Dec 2025 12:10:54 GMT

No problem @dvd_lg_bricks 🙂

Re: Questions About Workers and Executors Configuration in Databricks

dvd_lg_bricks — Tue, 09 Dec 2025 14:11:26 GMT

I mean: while we’re at it @szymon_dybczak or @Raman_Unifeye , is there a place where all available Databricks configuration parameters are documented? I have some pipelines that rely on special settings, such as changing the serializer, enabling Apache Arrow, and a few other uncommon configs.

Re: Questions About Workers and Executors Configuration in Databricks

Raman_Unifeye — Tue, 09 Dec 2025 14:29:53 GMT

perhaps you are looking for this - https://docs.databricks.com/aws/en/spark/conf

Re: Questions About Workers and Executors Configuration in Databricks

dvd_lg_bricks — Fri, 12 Dec 2025 11:21:36 GMT

Hello again,

I have a couple of questions regarding Databricks cluster configuration and best practices.

Are there any recommended best practices or guidelines for configuring Databricks clusters (e.g. sizing, cores per executor, memory settings, etc.) depending on the workload?

In on-premise Spark deployments, it is sometimes recommended to leave a certain number of CPU cores or a percentage of CPU/memory reserved for the operating system and the JVM (for example, not allocating 100% of resources to Spark executors).
Is there an equivalent recommendation or consideration in Databricks-managed environments, or is resource management fully handled by the platform?

Thanks in advance for your help.

Re: Questions About Workers and Executors Configuration in Databricks

Raman_Unifeye — Fri, 12 Dec 2025 15:23:58 GMT

Resource management for a cluster is fully handled by Databricks. Ideally, one would be more focused on the cluster sizing based on their workload type and set the autoscaling parameters for the cost/load balancing.

In our various recent implementation, we are moving more towards serverless to get away with the management overhead. Obviously, serverless do not fit for every use-case, so still need to work on hybrid method, having serverless-first approach.

Re: Questions About Workers and Executors Configuration in Databricks

szymon_dybczak — Fri, 12 Dec 2025 15:40:27 GMT

Hi @dvd_lg_bricks ,

There's a section about cluster configuration and tuning in Comprehensive Guide to Optimize Databricks, Spark and Delta Lake Workloads:

Comprehensive Guide to Optimize Data Workloads | Databricks

Re: Questions About Workers and Executors Configuration in Databricks

Abeshek — Fri, 12 Dec 2025 16:01:08 GMT

Your Databricks question about workers versus executors. Many teams encounter the same sizing and configuration issues when evaluating a migration. At Kanerika, we help companies plan cluster architecture, optimize Spark workloads, and avoid overspend during the move. If you want, I can share a quick sizing framework we use with clients. Would you be open to a brief 15-minute exchange next week?