Pool Max Capacity and Cluster Creation

[CodeInYellow]

Hello,

I have a theoretical question for which I have not been able to find a clear answer in the documentation.

When a cluster is created using an instance pool, what exactly is checked when the pool is asked to provide nodes?

More specifically, does the pool check whether:

• the maximum number of nodes configured for the new cluster is less than or equal to the number of nodes currently available in the pool (idle or not),

or whether:

• the maximum number of nodes configured for the new cluster is less than or equal to the number of nodes available in the pool (idle or not), after also accounting for the remaining capacity that could still be allocated to existing clusters already using the pool but not yet at their maximum size?

For example, let’s say we have a pool with a maximum capacity of 23 nodes.

• A first cluster is created with 6 nodes and a maximum of 12 nodes. The pool accepts the request and provides the initial 6 nodes.

• Then a second cluster, also configured with 6 nodes and a maximum of 12 nodes, is created.

Would the pool accept the second cluster because 23 - 6 = 17, which is still greater than 12?

Or would it reject it because 23 - 12 = 11, which is less than 12, meaning the first cluster’s unused scaling capacity is already effectively reserved?

Thank you in advance for your clarification.

[pradeep_singh]

Things work differently when you configure a cluster versus when the cluster is running and needs to autoscale.

During Configuration: It only checks if the cluster's max workers <= pool max capacity.

At Runtime: When the cluster needs to autoscale, it checks if the total pool instances (idle + in-use) + the requested instances <= pool max capacity.

If the pol is full the request will fail . 

In your case the creation of the cluster should be allowed . when it has to auto scale that when u might see issues when the total instances go beyond the pool size . 

[Louis_Frolio · Accepted solution]

Greetings @CodeInYellow , I did some research and here is what I found. 

Your first scenario is correct: the pool checks actual current usage, not possible future usage across attached clusters.

Using your example with a pool max capacity of 23:

1. Cluster 1 is created requesting 6 nodes. Pool usage becomes 6, leaving 17 available.

2. Cluster 2 is created requesting 6 nodes. The pool sees 17 available, so it allows the request.

What the pool does not do is reserve extra capacity for Cluster 1’s possible future scale-up to 12. There is no cross-cluster reservation mechanism.

The constraint shows up later at autoscale time. If both clusters try to scale to 12, that would require 24 total instances, which exceeds the pool max of 23. At that point, the request that pushes usage over the limit will fail with an INSTANCE_POOL_MAX_CAPACITY_FAILURE error.

One related nuance: if a single cluster’s max workers exceed the pool’s total max capacity, cluster creation will fail immediately. But that is a per-cluster validation check, not shared capacity reservation.

So the simplest way to think about it is:

“Do I have enough capacity right now for this request?”

not

“Do I have enough capacity for every cluster’s worst-case future scale at the same time?”

Hope that helps.

Cheers, Louis

---