cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How does cluster autoscaling work?

User16826992666
Valued Contributor

What determines when the cluster autoscaling activates to add and remove workers? Also, can it be adjusted?

1 ACCEPTED SOLUTION

Accepted Solutions

sajith_appukutt
Honored Contributor II

> What determines when the cluster autoscaling activates to add and remove workers

During scale-down, the service removes a worker only if it is idle and does not contain any shuffle data. This allows aggressive resizing without killing tasks or recomputing intermediate results . It also scales the cluster up aggressively in response to demand to keep responsiveness high without sacrificing efficiency. More details at https://databricks.com/blog/2018/05/02/introducing-databricks-optimized-auto-scaling.html

>Also, can it be adjusted?

Databricks offers two types of cluster node autoscaling: standard and optimized. Depending on the type, the parameters you could tune are

spark.databricks.aggressiveWindowDownS
spark.databricks.autoscaling.standardFirstStepUp

View solution in original post

1 REPLY 1

sajith_appukutt
Honored Contributor II

> What determines when the cluster autoscaling activates to add and remove workers

During scale-down, the service removes a worker only if it is idle and does not contain any shuffle data. This allows aggressive resizing without killing tasks or recomputing intermediate results . It also scales the cluster up aggressively in response to demand to keep responsiveness high without sacrificing efficiency. More details at https://databricks.com/blog/2018/05/02/introducing-databricks-optimized-auto-scaling.html

>Also, can it be adjusted?

Databricks offers two types of cluster node autoscaling: standard and optimized. Depending on the type, the parameters you could tune are

spark.databricks.aggressiveWindowDownS
spark.databricks.autoscaling.standardFirstStepUp