How the Scale up process done in the databricks cluster?

[Nandhini_Kumar]

For my AWS databricks cluster, i configured shared computer with 1min worker node and 3 max worker node, initailly only one worker node and driver node instance is created in the AWS console. 
Is there any rule set by databricks for scale up the next node like any threshold exceeds in the initial node(min node)?

How the scale up process done from one node to another node by databricks automatically?

[Kaniz_Fatma]

Hi @Nandhini_Kumar, 

1. Cluster Configuration:

   • When you create a Databricks cluster, you have several options for compute configuration. These choices impact performance, cost, and scalability.
   • Two primary types of computing are available:
     • All-purpose compute: Shared by multiple users, suitable for ad-hoc analysis, data exploration, or development.
     • Job compute: Used for operationalizing code after development. Job computes terminate when the job ends, reducing resource usage and cost.
   • You can also choose between single-node (for small workloads) and multi-node (for larger, distribute...¹.

2. Autoscaling:

   • Databricks provides an optimized autoscaling service that dynamically adjusts the number of workers based on load.
   • Here’s how it works:
     • When you provide a range for the number of workers, Databricks selects the appropriate number of workers needed for your job.
     • Autoscaling ensures efficient resource utilization without manual intervention.
     • Under low utilization, clusters can be scaled down aggressively while maintaining responsiveness.
     • Autoscaling helps balance cost and performance².

3. Thresholds and Scaling Rules:

   • Databricks does not have strict rules based on thresholds for scaling up from the initial minimum node.
   • However, you can configure autoscaling behavior based on your workload requirements.
   • For example, you can set rules to upscale when query wait times exceed a certain threshold or when load increases significantly.
   • Similarly, if the load decreases, Databricks can downscale the cluster to save resources³.

4. Visibility and Control:

   • Users do not directly start or stop compute instances. Instead, Databricks manages scaling.
   • The initial on-demand instances are immediately available to respond to user queries.
   • Enabling autoscaling allows the compute to scale up and down as needed¹.

In summary, Databricks optimizes cluster scaling by dynamically reallocating workers, ensuring efficient resource utilization, and maintaining responsiveness. While there isn’t a fixed rule for scaling thresholds, autoscaling adapts to workload demands automatically. 🚀🔍

For more detailed configuration options, you can refer to the official Databricks documentation¹.