https://community.databricks.com/t5/data-engineering/spark-configuration-parameter-for-cluster-downscaling/m-p/48148#M28255 <P>spark.databricks.aggressiveWindowDownS This parameter is designed to determine the frequency, in seconds, at which the cluster decides to downscale.</P><P>By adjusting this setting, you can fine-tune how rapidly clusters release workers. A higher value will result in the cluster holding onto workers longer before releasing them. The maximum limit for this parameter is set at 600 seconds.</P> Wed, 04 Oct 2023 13:43:43 GMT Hubert-Dudek 2023-10-04T13:43:43Z https://community.databricks.com/t5/data-engineering/spark-configuration-parameter-for-cluster-downscaling/m-p/48148#M28255 <P>spark.databricks.aggressiveWindowDownS This parameter is designed to determine the frequency, in seconds, at which the cluster decides to downscale.</P><P>By adjusting this setting, you can fine-tune how rapidly clusters release workers. A higher value will result in the cluster holding onto workers longer before releasing them. The maximum limit for this parameter is set at 600 seconds.</P> Wed, 04 Oct 2023 13:43:43 GMT https://community.databricks.com/t5/data-engineering/spark-configuration-parameter-for-cluster-downscaling/m-p/48148#M28255 Hubert-Dudek 2023-10-04T13:43:43Z https://community.databricks.com/t5/data-engineering/spark-configuration-parameter-for-cluster-downscaling/m-p/48157#M28256 <P>I wish there was a configuration to toggle upscaling behavior. I want the clusters to scale up only if the bottleneck is approaching 70% memory usage. Currently the autoscaling is only based on CPU not Memory (RAM).</P> Wed, 04 Oct 2023 14:08:12 GMT https://community.databricks.com/t5/data-engineering/spark-configuration-parameter-for-cluster-downscaling/m-p/48157#M28256 Haiyangl104 2023-10-04T14:08:12Z