cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Administration & Architecture
Explore discussions on Databricks administration, deployment strategies, and architectural best practices. Connect with administrators and architects to optimize your Databricks environment for performance, scalability, and security.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Use different instance types in pools

User16826992666
Valued Contributor

I am wondering if it's possible to create a pool that has a mix of instance types in it?

1 ACCEPTED SOLUTION

Accepted Solutions

User16783853906
Contributor III

We started the Private Preview of AWS EC2 Fleet from early 2021. AWS EC2 Fleet feature allows customers to create Databricks pools and get VMs of multiple AZs and multiple instance families & sizes. When clusters are created from the Databricks EC2 fleet pools, they gain the benefit of even lower EC2 spot price and more instance availability. Specifically:

  • Clusters created from EC2 Fleet pool will get VMs from the cheapest AZ within your pool definition.
  • They will also get VMs of the cheapest type within your pool definition.
  • When clusters experience a spot kill, instead of falling back to on-demand VMs, the pool will look for another spot VM from either a different AZ or another instance type.

View solution in original post

5 REPLIES 5

brickster_2018
Databricks Employee
Databricks Employee

Pools are a homogenous collection of instances. So it's not possible to mix instance types in it.

However, it's possible to create two different pools and use one for the driver instance and the other for the executor instance. This feature is very useful as in most of cases the resource requirement of the Driver is different from the executor

@harikrishnan kunhumveettilโ€‹  This answer needs to be updated I guess, we support different instance types with pools in AWS env.

User16783853906
Contributor III

We started the Private Preview of AWS EC2 Fleet from early 2021. AWS EC2 Fleet feature allows customers to create Databricks pools and get VMs of multiple AZs and multiple instance families & sizes. When clusters are created from the Databricks EC2 fleet pools, they gain the benefit of even lower EC2 spot price and more instance availability. Specifically:

  • Clusters created from EC2 Fleet pool will get VMs from the cheapest AZ within your pool definition.
  • They will also get VMs of the cheapest type within your pool definition.
  • When clusters experience a spot kill, instead of falling back to on-demand VMs, the pool will look for another spot VM from either a different AZ or another instance type.

abagshaw
New Contributor III

AWS Fleet instance types are now GA and available for clusters and instance pools. You can see more details here: https://community.databricks.com/t5/user/viewprofilepage/user-id/83278

abagshaw
New Contributor III

AWS Fleet instance types are now GA and available for clusters and instance pools. You can see more details here: https://docs.databricks.com/compute/aws-fleet-instances.html

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group