https://community.databricks.com/t5/data-engineering/how-long-does-a-task-have-to-be-in-the-queue-before-the-cluster/m-p/25116#M17423 Fri, 11 Jun 2021 04:16:12 GMT Anonymous 2021-06-11T04:16:12Z https://community.databricks.com/t5/data-engineering/how-long-does-a-task-have-to-be-in-the-queue-before-the-cluster/m-p/25116#M17423 Fri, 11 Jun 2021 04:16:12 GMT https://community.databricks.com/t5/data-engineering/how-long-does-a-task-have-to-be-in-the-queue-before-the-cluster/m-p/25116#M17423 Anonymous 2021-06-11T04:16:12Z https://community.databricks.com/t5/data-engineering/how-long-does-a-task-have-to-be-in-the-queue-before-the-cluster/m-p/25117#M17424 <P>There are two types of auto scaling in Databricks: Standard and Optimized. In both scenarios when tasks are submitted the cluster will begin scaling to execute as many of them in parallel immediately.</P><P></P><P></P><P>Scaling down is different. In optimized autoscaling down will occur if the cluster is underutilized over the last 40 seconds on job clusters or 150 seconds on all-purpose clusters.</P><P></P><P></P><P>Standard autoscaling down will occur only when the cluster is completely idle and underutilized for the last 10 minutes. Standard scaling is exponential and going down will start with 1 node (scaling up starts with 8 nodes).</P><P></P><P></P><P>Check out the documentation <A href="https://docs.databricks.com/clusters/configure.html#cluster-size-and-autoscaling" alt="https://docs.databricks.com/clusters/configure.html#cluster-size-and-autoscaling" target="_blank">here</A>.</P> Fri, 11 Jun 2021 08:46:00 GMT https://community.databricks.com/t5/data-engineering/how-long-does-a-task-have-to-be-in-the-queue-before-the-cluster/m-p/25117#M17424 Ryan_Chynoweth 2021-06-11T08:46:00Z