topic How to implement automatic scaling of cluster size in Serverless Warehouse in Warehousing & Analytics

How to implement automatic scaling of cluster size in Serverless Warehouse

fishingrod — Fri, 14 Feb 2025 14:36:23 GMT

I would like to know if the cluster size of a Serverless Warehouse can automatically scale up and down, and what determines the number of workers used when executing queries. Does it use all workers within the cluster size fixedly, or does it use part or all of the workers in a cluster according to the query plan?

Re: How to implement automatic scaling of cluster size in Serverless Warehouse

Alberto_Umana — Fri, 14 Feb 2025 14:59:08 GMT

Hello @fishingrod,

Yes, the cluster size of a Serverless Warehouse can automatically scale up and down. The autoscaling behavior is managed by Intelligent Workload Management (IWM), which dynamically adjusts the compute resources based on the workload demands. This system uses machine learning models to predict the resource needs of incoming queries and monitors the warehouse’s available compute capacity in real time.

The number of workers used when executing queries is not fixed. Instead, the system scales the number of resources to accommodate varying demands. The autoscaling mechanism ensures that compute resources are rapidly upscaled to maintain low latency and quickly downscaled to minimize costs when demand is low. This dynamic management allows the warehouse to handle large numbers of queries efficiently and cost-effectively.

Please see: https://docs.databricks.com/en/compute/sql-warehouse/warehouse-behavior.html

Re: How to implement automatic scaling of cluster size in Serverless Warehouse

fishingrod — Sat, 15 Feb 2025 09:07:52 GMT

So what is the purpose of the `cluster size` I set when I created the Serverless Warehouse? Isn’t it automatically adjusted?

Re: How to implement automatic scaling of cluster size in Serverless Warehouse

Takuya-Omi — Sat, 15 Feb 2025 14:04:01 GMT

@fishingrod

My understanding is that Intelligent Workload Management (IWM) in Serverless SQL Warehouses adjusts the number of clusters, but it does not automatically scale the cluster size.

This means that if you need to improve the execution performance of individual SQL queries, you will need to explicitly specify the cluster size yourself.