Understanding Autoscaling in Databricks: Under What Conditions Does Spark Add a New Worker Node?

[h_h_ak]

I’m currently working with Databricks autoscaling configurations and trying to better understand how Spark decides when to spin up additional worker nodes. My cluster has a minimum of one worker and can scale up to five. I know that tasks are assigned to cores and that if more tasks are queued than available cores, Spark may consider adding a new worker—assuming autoscaling is enabled. But what specific conditions or metrics does Spark use to trigger the autoscaling event?

For example:

• Is it based solely on the number of pending tasks in the scheduler queue?
• Does it consider task completion times, memory usage, or CPU utilization on existing workers?
• How quickly does autoscaling react once these conditions are met?

A practical scenario: If I have a single worker with 8 cores and I have more tasks than cores for a prolonged period, will Spark immediately add another worker or does it wait for some threshold of sustained load?

I’d appreciate insights from anyone who has worked with Databricks autoscaling in production. Any reference to official documentation or real-world examples of how Spark conditions must be met before a new worker is allocated would be very helpful.

[17abhishek]

Databricks autoscaling is designed to dynamically adjust the number of worker nodes in a cluster based on workload demand, optimizing resource utilization and cost. Understanding the conditions under which Spark triggers autoscaling requires insight into how Databricks monitors and interprets the workload.

[filipniziol]

Hi @h_h_ak ,

Short Answer:

• Autoscaling primarily depends on the number of pending tasks.
• Workspaces on the Premium plan use optimized autoscaling, while those on the Standard plan use standard autoscaling.

Long Answer:

Databricks autoscaling responds mainly to sustained backlogs of unscheduled tasks rather than CPU or memory usage alone. If the number of pending tasks consistently exceeds your current cluster capacity—meaning more tasks are queued than available cores can handle—Databricks will consider adding a new worker node.

Key Points:

• Pending Tasks as the Main Trigger: Autoscaling monitors how many tasks remain queued. Persistent queues indicate that existing workers can’t keep up, prompting additional workers.
• Not Instantaneous, But Sustained Load: Spark waits to confirm that the increased demand isn’t just a short-lived spike. Only after tasks remain pending for a threshold period does scaling occur.
• Indirect Role of CPU/Memory Utilization: While CPU/memory affect task completion speed, autoscaling decisions are based on task queues rather than these metrics directly.
• Timing and Reaction: Adding a new worker typically takes a minute or so, ensuring scaling responds to stable workload increases rather than momentary fluctuations.

Useful Links:

• https://learn.microsoft.com/en-us/azure/databricks/compute/configure#autoscaling
• https://www.databricks.com/blog/2018/05/02/introducing-databricks-optimized-auto-scaling.html
• https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation

[h_h_ak]

Hi @filipniziol,

Great summary, thanks!

It would interest me to know what the limits are in more detail. For example:

• How many pending tasks are needed to trigger a new worker or cluster?
• How long does the CPU utilization need to be above a certain threshold (e.g., > XX%) before scaling occurs?

Are there specific thresholds or configurable parameters that influence these decisions?

Thanks again for the clarification!