How to implement automatic scaling of cluster size in Serverless Warehouse

[fishingrod]

I would like to know if the cluster size of a Serverless Warehouse can automatically scale up and down, and what determines the number of workers used when executing queries. Does it use all workers within the cluster size fixedly, or does it use part or all of the workers in a cluster according to the query plan?

[Alberto_Umana]

Hello @fishingrod,

Yes, the cluster size of a Serverless Warehouse can automatically scale up and down. The autoscaling behavior is managed by Intelligent Workload Management (IWM), which dynamically adjusts the compute resources based on the workload demands. This system uses machine learning models to predict the resource needs of incoming queries and monitors the warehouse’s available compute capacity in real time.

The number of workers used when executing queries is not fixed. Instead, the system scales the number of resources to accommodate varying demands. The autoscaling mechanism ensures that compute resources are rapidly upscaled to maintain low latency and quickly downscaled to minimize costs when demand is low. This dynamic management allows the warehouse to handle large numbers of queries efficiently and cost-effectively.

Please see: https://docs.databricks.com/en/compute/sql-warehouse/warehouse-behavior.html

[fishingrod]

So what is the purpose of the `cluster size` I set when I created the Serverless Warehouse? Isn’t it automatically adjusted?

[Takuya-Omi]

@fishingrod 

My understanding is that Intelligent Workload Management (IWM) in Serverless SQL Warehouses adjusts the number of clusters, but it does not automatically scale the cluster size.

This means that if you need to improve the execution performance of individual SQL queries, you will need to explicitly specify the cluster size yourself.