What’s the largest cluster/maximum number of cores you can spin up in the Databricks environment?

[sean_owen]

There isn't a Databricks maximum, but, you may be limited by the quota of cores you are allowed to access in the cloud according to your cloud provider account's limits.

[User16826994223]

Generally it is limited by cloud provider, initially yo get around 350 cores that can be increased by request to cloud vendor, Till now I have seen 1000 cores and it can go much more

In addition to subscription limits, the total capacity of clusters in each workspace is a function of the masks used for the workspace's enclosing Vnet and the pair of subnets associated with each cluster in the workspace. The masks can be changed if you use the Bring Your Own Vnet feature as it gives you more control over the networking layout.

Worth noting that in Azure

• Each cluster node requires 1 Public IP and 2 Private IPs
• These IPs are logically grouped into 2 subnets named “public” and “private”
• For a desired cluster size of X: number of Public IPs = X, number of Private IPs = 2X