How does cluster autoscaling work?

User16826992666 — Wed, 16 Jun 2021 04:03:50 GMT

What determines when the cluster autoscaling activates to add and remove workers? Also, can it be adjusted?

Re: How does cluster autoscaling work?

sajith_appukutt — Thu, 17 Jun 2021 22:26:46 GMT

> What determines when the cluster autoscaling activates to add and remove workers

During scale-down, the service removes a worker only if it is idle and does not contain any shuffle data. This allows aggressive resizing without killing tasks or recomputing intermediate results . It also scales the cluster up aggressively in response to demand to keep responsiveness high without sacrificing efficiency. More details at https://databricks.com/blog/2018/05/02/introducing-databricks-optimized-auto-scaling.html

>Also, can it be adjusted?

Databricks offers two types of cluster node autoscaling: standard and optimized. Depending on the type, the parameters you could tune are

spark.databricks.aggressiveWindowDownS
spark.databricks.autoscaling.standardFirstStepUp

topic How does cluster autoscaling work? in Data Engineering

How does cluster autoscaling work?

Re: How does cluster autoscaling work?