Harun
Honored Contributor

I agree with the @Retko Okter

To support the second point, find the below explanation,​

Optimized autoscaling

  • Firstly, scales up from min to max in 2 steps.
  • Secondly, can scale down even if the cluster is not idle by looking at shuffle file state.
  • Thirdly, scales down based on a percentage of current nodes.
  • And, on job clusters, scales down if the cluster is underutilized over the last 40 seconds.
  • Lastly, on all-purpose clusters, scales down if the cluster is underutilized over the last 150 seconds.

Standard autoscaling

  • Firstly, starts with adding 8 nodes. Thereafter, scales up exponentially, but can take many steps to reach the max.
  • Secondly, scales down only when the cluster is completely idle and it has been underutilized for the last 10 minutes.
  • Lastly, scales down exponentially, starting with 1 node.