cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How long does a task have to be in the queue before the cluster autoscales?:

Anonymous
Not applicable
 
1 ACCEPTED SOLUTION

Accepted Solutions

Ryan_Chynoweth
Esteemed Contributor

There are two types of auto scaling in Databricks: Standard and Optimized. In both scenarios when tasks are submitted the cluster will begin scaling to execute as many of them in parallel immediately.

Scaling down is different. In optimized autoscaling down will occur if the cluster is underutilized over the last 40 seconds on job clusters or 150 seconds on all-purpose clusters.

Standard autoscaling down will occur only when the cluster is completely idle and underutilized for the last 10 minutes. Standard scaling is exponential and going down will start with 1 node (scaling up starts with 8 nodes).

Check out the documentation here.

View solution in original post

1 REPLY 1

Ryan_Chynoweth
Esteemed Contributor

There are two types of auto scaling in Databricks: Standard and Optimized. In both scenarios when tasks are submitted the cluster will begin scaling to execute as many of them in parallel immediately.

Scaling down is different. In optimized autoscaling down will occur if the cluster is underutilized over the last 40 seconds on job clusters or 150 seconds on all-purpose clusters.

Standard autoscaling down will occur only when the cluster is completely idle and underutilized for the last 10 minutes. Standard scaling is exponential and going down will start with 1 node (scaling up starts with 8 nodes).

Check out the documentation here.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group