Spark Configuration Parameter for Cluster Downscaling

[Hubert-Dudek]

spark.databricks.aggressiveWindowDownS This parameter is designed to determine the frequency, in seconds, at which the cluster decides to downscale.

By adjusting this setting, you can fine-tune how rapidly clusters release workers. A higher value will result in the cluster holding onto workers longer before releasing them. The maximum limit for this parameter is set at 600 seconds.

[Haiyangl104]

I wish there was a configuration to toggle upscaling behavior. I want the clusters to scale up only if the bottleneck is approaching 70% memory usage. Currently the autoscaling is only based on CPU not Memory (RAM).