cancel
Showing results for 
Search instead for 
Did you mean: 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results for 
Search instead for 
Did you mean: 

Use different instance types in pools

User16826992666
Valued Contributor

I am wondering if it's possible to create a pool that has a mix of instance types in it?

1 ACCEPTED SOLUTION

Accepted Solutions

User16783853906
Contributor III

We started the Private Preview of AWS EC2 Fleet from early 2021. AWS EC2 Fleet feature allows customers to create Databricks pools and get VMs of multiple AZs and multiple instance families & sizes. When clusters are created from the Databricks EC2 fleet pools, they gain the benefit of even lower EC2 spot price and more instance availability. Specifically:

  • Clusters created from EC2 Fleet pool will get VMs from the cheapest AZ within your pool definition.
  • They will also get VMs of the cheapest type within your pool definition.
  • When clusters experience a spot kill, instead of falling back to on-demand VMs, the pool will look for another spot VM from either a different AZ or another instance type.

View solution in original post

5 REPLIES 5

brickster_2018
Esteemed Contributor
Esteemed Contributor

Pools are a homogenous collection of instances. So it's not possible to mix instance types in it.

However, it's possible to create two different pools and use one for the driver instance and the other for the executor instance. This feature is very useful as in most of cases the resource requirement of the Driver is different from the executor

@harikrishnan kunhumveettil​  This answer needs to be updated I guess, we support different instance types with pools in AWS env.

User16783853906
Contributor III

We started the Private Preview of AWS EC2 Fleet from early 2021. AWS EC2 Fleet feature allows customers to create Databricks pools and get VMs of multiple AZs and multiple instance families & sizes. When clusters are created from the Databricks EC2 fleet pools, they gain the benefit of even lower EC2 spot price and more instance availability. Specifically:

  • Clusters created from EC2 Fleet pool will get VMs from the cheapest AZ within your pool definition.
  • They will also get VMs of the cheapest type within your pool definition.
  • When clusters experience a spot kill, instead of falling back to on-demand VMs, the pool will look for another spot VM from either a different AZ or another instance type.

abagshaw
New Contributor III

AWS Fleet instance types are now GA and available for clusters and instance pools. You can see more details here: https://community.databricks.com/t5/user/viewprofilepage/user-id/83278

abagshaw
New Contributor III

AWS Fleet instance types are now GA and available for clusters and instance pools. You can see more details here: https://docs.databricks.com/compute/aws-fleet-instances.html

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!