How the Scale up process done in the databricks cluster?

Nandhini_Kumar — Mon, 08 Apr 2024 10:51:09 GMT

For my AWS databricks cluster, i configured shared computer with 1min worker node and 3 max worker node, initailly only one worker node and driver node instance is created in the AWS console.
Is there any rule set by databricks for scale up the next node like any threshold exceeds in the initial node(min node)?

How the scale up process done from one node to another node by databricks automatically?

Re: How the Scale up process done in the databricks cluster?

NandiniN — Thu, 09 Jan 2025 07:55:49 GMT

Databricks uses autoscaling to manage the number of worker nodes in a cluster based on the workload. When you configure a cluster with a minimum and maximum number of worker nodes, Databricks automatically adjusts the number of workers within this range based on the demand.

The autoscaling process works as follows:

Initial Setup: When you start the cluster, it begins with the minimum number of worker nodes specified (in your case, 1 worker node).
Scaling Up: Databricks monitors the workload on the cluster. If the workload increases and the current number of worker nodes is insufficient to handle the load, Databricks will add more worker nodes up to the maximum limit specified (3 worker nodes in your case). The scaling up process is triggered by the demand for resources such as CPU and memory.
Scaling Down: Similarly, if the workload decreases and the current number of worker nodes is more than necessary, Databricks will remove worker nodes to save costs, but it will not go below the minimum number of worker nodes specified.

The specific thresholds and rules for scaling up and down are managed by Databricks' autoscaling algorithms. For example, optimized autoscaling in Databricks can scale up from the minimum to the maximum number of nodes in two steps and can scale down based on the underutilization of nodes over a certain period (e.g., 40 seconds for job compute and 150 seconds for all-purpose compute).

Doc - https://docs.databricks.com/en/compute/configure.html#how-autoscaling-behaves

topic Re: How the Scale up process done in the databricks cluster? in Get Started Discussions

How the Scale up process done in the databricks cluster?

Re: How the Scale up process done in the databricks cluster?