https://community.databricks.com/t5/missing-questionpost/why-doesn-t-my-job-cluster-scale-down/m-p/2773#M17 <P>Hi,</P><P>Running on Databricks-AWS, I have a job running on a cluster with 3 workers, 2-cores each (r6i.large), with <B>autoscaling enabled</B>.</P><P><U>The Spark job has two stages</U>: </P><P><B>(1)</B> highly parallelizable, cpu-intensive stage. This stage takes 15 minutes.</P><P><B>(2)</B> a non-parallelizable stage (only a single partition, so a single spark task). This stage takes 45 minutes.</P><P></P><P>In the first stage, the cluster scales up from 1 worker to 3, and all 3 workers (6 cores) are fully utilized for the duration of the stage (15 minutes). Then, in the second stage, only a single worker node is active for the entire 45 minutes, but databricks does not scale down my cluster and I have two nodes completely idle for 45 minutes.</P><P></P><P>Any idea why that is, and how I can utilize autoscaling to be more cost efficient in this type of job?</P><P>Thanks!</P> Wed, 21 Jun 2023 12:48:27 GMT 804925 2023-06-21T12:48:27Z https://community.databricks.com/t5/missing-questionpost/why-doesn-t-my-job-cluster-scale-down/m-p/2773#M17 <P>Hi,</P><P>Running on Databricks-AWS, I have a job running on a cluster with 3 workers, 2-cores each (r6i.large), with <B>autoscaling enabled</B>.</P><P><U>The Spark job has two stages</U>: </P><P><B>(1)</B> highly parallelizable, cpu-intensive stage. This stage takes 15 minutes.</P><P><B>(2)</B> a non-parallelizable stage (only a single partition, so a single spark task). This stage takes 45 minutes.</P><P></P><P>In the first stage, the cluster scales up from 1 worker to 3, and all 3 workers (6 cores) are fully utilized for the duration of the stage (15 minutes). Then, in the second stage, only a single worker node is active for the entire 45 minutes, but databricks does not scale down my cluster and I have two nodes completely idle for 45 minutes.</P><P></P><P>Any idea why that is, and how I can utilize autoscaling to be more cost efficient in this type of job?</P><P>Thanks!</P> Wed, 21 Jun 2023 12:48:27 GMT https://community.databricks.com/t5/missing-questionpost/why-doesn-t-my-job-cluster-scale-down/m-p/2773#M17 804925 2023-06-21T12:48:27Z https://community.databricks.com/t5/missing-questionpost/why-doesn-t-my-job-cluster-scale-down/m-p/2774#M18 <P>Hi @Yoav Ben​&nbsp;</P><P></P><P>Great to meet you, and thanks for your question! </P><P></P><P>Let's see if your peers in the community have an answer to your question. Thanks.</P> Thu, 22 Jun 2023 05:03:42 GMT https://community.databricks.com/t5/missing-questionpost/why-doesn-t-my-job-cluster-scale-down/m-p/2774#M18 Anonymous 2023-06-22T05:03:42Z