What’s the largest cluster/maximum number of cores you can spin up in the Databricks environment?

User16790091296 — Fri, 28 May 2021 19:11:00 GMT

Re: What’s the largest cluster/maximum number of cores you can spin up in the Databricks environment?

sean_owen — Fri, 18 Jun 2021 00:03:23 GMT

There isn't a Databricks maximum, but, you may be limited by the quota of cores you are allowed to access in the cloud according to your cloud provider account's limits.

Re: What’s the largest cluster/maximum number of cores you can spin up in the Databricks environment?

User16826994223 — Fri, 18 Jun 2021 08:06:21 GMT

Generally it is limited by cloud provider, initially yo get around 350 cores that can be increased by request to cloud vendor, Till now I have seen 1000 cores and it can go much more

In addition to subscription limits, the total capacity of clusters in each workspace is a function of the masks used for the workspace's enclosing Vnet and the pair of subnets associated with each cluster in the workspace. The masks can be changed if you use the Bring Your Own Vnet feature as it gives you more control over the networking layout.

Worth noting that in Azure

Each cluster node requires 1 Public IP and 2 Private IPs
These IPs are logically grouped into 2 subnets named “public” and “private”
For a desired cluster size of X: number of Public IPs = X, number of Private IPs = 2X

topic Re: What’s the largest cluster/maximum number of cores you can spin up in the Databricks environment? in Data Engineering

What’s the largest cluster/maximum number of cores you can spin up in the Databricks environment?

Re: What’s the largest cluster/maximum number of cores you can spin up in the Databricks environment?

Re: What’s the largest cluster/maximum number of cores you can spin up in the Databricks environment?