Use different instance types in pools

[User16826992666]

I am wondering if it's possible to create a pool that has a mix of instance types in it?

[User16869510359]

Pools are a homogenous collection of instances. So it's not possible to mix instance types in it.

However, it's possible to create two different pools and use one for the driver instance and the other for the executor instance. This feature is very useful as in most of cases the resource requirement of the Driver is different from the executor

[User16783853906]

@harikrishnan kunhumveettil​  This answer needs to be updated I guess, we support different instance types with pools in AWS env.

[User16783853906 · Accepted solution]

We started the Private Preview of AWS EC2 Fleet from early 2021. AWS EC2 Fleet feature allows customers to create Databricks pools and get VMs of multiple AZs and multiple instance families & sizes. When clusters are created from the Databricks EC2 fleet pools, they gain the benefit of even lower EC2 spot price and more instance availability. Specifically:

• Clusters created from EC2 Fleet pool will get VMs from the cheapest AZ within your pool definition.
• They will also get VMs of the cheapest type within your pool definition.
• When clusters experience a spot kill, instead of falling back to on-demand VMs, the pool will look for another spot VM from either a different AZ or another instance type.

[abagshaw]

AWS Fleet instance types are now GA and available for clusters and instance pools. You can see more details here: https://community.databricks.com/t5/user/viewprofilepage/user-id/83278

[abagshaw]

AWS Fleet instance types are now GA and available for clusters and instance pools. You can see more details here: https://docs.databricks.com/compute/aws-fleet-instances.html