What’s the largest cluster/maximum number of cores you can spin up in the Databricks environment?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-28-2021 12:11 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-17-2021 05:03 PM
There isn't a Databricks maximum, but, you may be limited by the quota of cores you are allowed to access in the cloud according to your cloud provider account's limits.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-18-2021 01:06 AM
Generally it is limited by cloud provider, initially yo get around 350 cores that can be increased by request to cloud vendor, Till now I have seen 1000 cores and it can go much more
In addition to subscription limits, the total capacity of clusters in each workspace is a function of the masks used for the workspace's enclosing Vnet and the pair of subnets associated with each cluster in the workspace. The masks can be changed if you use the Bring Your Own Vnet feature as it gives you more control over the networking layout.
Worth noting that in Azure
- Each cluster node requires 1 Public IP and 2 Private IPs
- These IPs are logically grouped into 2 subnets named “public” and “private”
- For a desired cluster size of X: number of Public IPs = X, number of Private IPs = 2X