topic Re: Use different instance types in pools in Administration & Architecture

Use different instance types in pools

User16826992666 — Tue, 18 Mar 2025 16:57:10 GMT

I am wondering if it's possible to create a pool that has a mix of instance types in it?

Re: Use different instance types in pools

brickster_2018 — Thu, 24 Jun 2021 23:30:01 GMT

Pools are a homogenous collection of instances. So it's not possible to mix instance types in it.

However, it's possible to create two different pools and use one for the driver instance and the other for the executor instance. This feature is very useful as in most of cases the resource requirement of the Driver is different from the executor

Re: Use different instance types in pools

User16783853906 — Fri, 25 Jun 2021 17:27:06 GMT

We started the Private Preview of AWS EC2 Fleet from early 2021. AWS EC2 Fleet feature allows customers to create Databricks pools and get VMs of multiple AZs and multiple instance families & sizes. When clusters are created from the Databricks EC2 fleet pools, they gain the benefit of even lower EC2 spot price and more instance availability. Specifically:

Clusters created from EC2 Fleet pool will get VMs from the cheapest AZ within your pool definition.
They will also get VMs of the cheapest type within your pool definition.
When clusters experience a spot kill, instead of falling back to on-demand VMs, the pool will look for another spot VM from either a different AZ or another instance type.

Re: Use different instance types in pools

User16783853906 — Fri, 25 Jun 2021 17:29:55 GMT

@harikrishnan kunhumveettil This answer needs to be updated I guess, we support different instance types with pools in AWS env.

Re: Use different instance types in pools

abagshaw — Tue, 27 Jun 2023 18:41:14 GMT

AWS Fleet instance types are now GA and available for clusters and instance pools. You can see more details here: https://community.databricks.com/t5/user/viewprofilepage/user-id/83278

Re: Use different instance types in pools

abagshaw — Tue, 27 Jun 2023 18:51:22 GMT

AWS Fleet instance types are now GA and available for clusters and instance pools. You can see more details here: https://docs.databricks.com/compute/aws-fleet-instances.html