https://community.databricks.com/t5/data-engineering/databricks-cluster-autoscaling/m-p/6425#M2575 <P>@Deepak Bhatt​&nbsp;:</P><P></P><P>Autoscaling in Databricks can have a significant impact on cluster cost, as it allows the cluster to dynamically add or remove workers based on the workload.</P><P>In the scenario you described, if the active worker count is consistently at 3, then the cluster will only be billed for 3 workers most of the time, regardless of the maximum number of workers set for the cluster. However, if there are occasional spikes in workload that require additional workers, then the cluster may temporarily scale up to meet the demand and incur additional costs during those periods.</P><P>The cost of autoscaling depends on the instance type and the duration of the scaling events. For example, adding or removing a worker may take a few minutes to complete, and during that time, the cluster will incur costs for the additional worker(s) even if they are not fully utilized. Additionally, larger instance types will have higher hourly costs than smaller ones, so using autoscaling with larger instances may result in higher costs.</P><P>To minimize costs with autoscaling, it's important to monitor cluster usage and adjust the minimum and maximum worker counts based on the workload patterns. Setting the minimum workers to the average number of workers needed during low periods can help to reduce costs, while still allowing for scaling up during high-demand periods. Similarly, setting the maximum workers to a reasonable level that meets the needs of the workload can help to avoid unnecessary costs associated with scaling up beyond what is necessary.</P> Fri, 07 Apr 2023 02:00:14 GMT Anonymous 2023-04-07T02:00:14Z https://community.databricks.com/t5/data-engineering/databricks-cluster-autoscaling/m-p/6424#M2574 <P>Hello All,</P><P></P><P>Could anyone please suggest impact of Autoscaling in cluster cost ?</P><P>Suppose if I have a cluster where min worker is 2 and max is 10 but most of the time active worker are 3 so the cluster will be billed for only 3 workers or for 10 worker(though most of time 7 worker are ideal).</P><P></P><P>Thanks,</P><P>Deepak</P> Tue, 04 Apr 2023 17:55:57 GMT https://community.databricks.com/t5/data-engineering/databricks-cluster-autoscaling/m-p/6424#M2574 Mr__D 2023-04-04T17:55:57Z https://community.databricks.com/t5/data-engineering/databricks-cluster-autoscaling/m-p/6425#M2575 <P>@Deepak Bhatt​&nbsp;:</P><P></P><P>Autoscaling in Databricks can have a significant impact on cluster cost, as it allows the cluster to dynamically add or remove workers based on the workload.</P><P>In the scenario you described, if the active worker count is consistently at 3, then the cluster will only be billed for 3 workers most of the time, regardless of the maximum number of workers set for the cluster. However, if there are occasional spikes in workload that require additional workers, then the cluster may temporarily scale up to meet the demand and incur additional costs during those periods.</P><P>The cost of autoscaling depends on the instance type and the duration of the scaling events. For example, adding or removing a worker may take a few minutes to complete, and during that time, the cluster will incur costs for the additional worker(s) even if they are not fully utilized. Additionally, larger instance types will have higher hourly costs than smaller ones, so using autoscaling with larger instances may result in higher costs.</P><P>To minimize costs with autoscaling, it's important to monitor cluster usage and adjust the minimum and maximum worker counts based on the workload patterns. Setting the minimum workers to the average number of workers needed during low periods can help to reduce costs, while still allowing for scaling up during high-demand periods. Similarly, setting the maximum workers to a reasonable level that meets the needs of the workload can help to avoid unnecessary costs associated with scaling up beyond what is necessary.</P> Fri, 07 Apr 2023 02:00:14 GMT https://community.databricks.com/t5/data-engineering/databricks-cluster-autoscaling/m-p/6425#M2575 Anonymous 2023-04-07T02:00:14Z