For sql classic, does number of driver also increase when auto scaled up?

[alexw4561]

I know for spark clusters, the number of driver will remain the same and only the number of workers increases when auto scaling. But for sql classic warehouse I'm a bit confused, since it's scaling number of cluster not driver.

[szymon_dybczak]

Hi @alexw4561 ,

I would assume that. They don’t scale by resizing a single cluster. Instead, they scale by adding or removing whole clusters. Each of those clusters has its own driver and workers. So if you configure min = 1 cluster, max = 10 clusters, Databricks can spin up to 10 separate clusters. The SQL Warehouse load balancer routes queries to one of these clusters. This is horizontal scaling at the cluster level.

[jebouwman]

Hi @alexw4561 , 

@szymon_dybczak is indeed right as this page explains: https://docs.databricks.com/aws/en/compute/sql-warehouse/warehouse-behavior#queuing-and-autoscaling-...

We had the same question in the past and found out it indeed just adds an extra cluster.

[nayan_wylde]

SQL Classic Warehouses:

• Scaling is done by adding or removing entire clusters, not just workers.
• Each cluster has its own driver and set of workers.
• So when autoscaling happens, more clusters (each with a driver + workers) are added to handle concurrent query load.

Suppose if you are creating a M size cluster and you set the min scale at 1 and max at 5. The calculation will be:  

• At low load, you run 1 cluster → 1 driver + 8 workers.
• At high load, Databricks may scale up to 5 clusters → 5 drivers + 40 workers.