- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-15-2021 09:03 PM
What determines when the cluster autoscaling activates to add and remove workers? Also, can it be adjusted?
- Labels:
-
Cluster
-
Cluster Autoscaling
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-17-2021 03:26 PM
> What determines when the cluster autoscaling activates to add and remove workers
During scale-down, the service removes a worker only if it is idle and does not contain any shuffle data. This allows aggressive resizing without killing tasks or recomputing intermediate results . It also scales the cluster up aggressively in response to demand to keep responsiveness high without sacrificing efficiency. More details at https://databricks.com/blog/2018/05/02/introducing-databricks-optimized-auto-scaling.html
>Also, can it be adjusted?
Databricks offers two types of cluster node autoscaling: standard and optimized. Depending on the type, the parameters you could tune are
spark.databricks.aggressiveWindowDownS
spark.databricks.autoscaling.standardFirstStepUp
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-17-2021 03:26 PM
> What determines when the cluster autoscaling activates to add and remove workers
During scale-down, the service removes a worker only if it is idle and does not contain any shuffle data. This allows aggressive resizing without killing tasks or recomputing intermediate results . It also scales the cluster up aggressively in response to demand to keep responsiveness high without sacrificing efficiency. More details at https://databricks.com/blog/2018/05/02/introducing-databricks-optimized-auto-scaling.html
>Also, can it be adjusted?
Databricks offers two types of cluster node autoscaling: standard and optimized. Depending on the type, the parameters you could tune are
spark.databricks.aggressiveWindowDownS
spark.databricks.autoscaling.standardFirstStepUp

