cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How long does a task have to be in the queue before the cluster autoscales?:

Anonymous
Not applicable
 
1 ACCEPTED SOLUTION

Accepted Solutions

Ryan_Chynoweth
Honored Contributor III

There are two types of auto scaling in Databricks: Standard and Optimized. In both scenarios when tasks are submitted the cluster will begin scaling to execute as many of them in parallel immediately.

Scaling down is different. In optimized autoscaling down will occur if the cluster is underutilized over the last 40 seconds on job clusters or 150 seconds on all-purpose clusters.

Standard autoscaling down will occur only when the cluster is completely idle and underutilized for the last 10 minutes. Standard scaling is exponential and going down will start with 1 node (scaling up starts with 8 nodes).

Check out the documentation here.

View solution in original post

1 REPLY 1

Ryan_Chynoweth
Honored Contributor III

There are two types of auto scaling in Databricks: Standard and Optimized. In both scenarios when tasks are submitted the cluster will begin scaling to execute as many of them in parallel immediately.

Scaling down is different. In optimized autoscaling down will occur if the cluster is underutilized over the last 40 seconds on job clusters or 150 seconds on all-purpose clusters.

Standard autoscaling down will occur only when the cluster is completely idle and underutilized for the last 10 minutes. Standard scaling is exponential and going down will start with 1 node (scaling up starts with 8 nodes).

Check out the documentation here.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.