topic Re: Support for multiple EC2 instance types in a worker pool in Data Engineering

Support for multiple EC2 instance types in a worker pool

JPKC — Wed, 01 Mar 2023 14:06:20 GMT

As per this thread Databricks now integrates with EC2 CreateFleet API that allows customers to create Databricks pools and get EC2 instances from multiple AZs and multiple instance families & sizes. However, in the Databricks UI you can not select more than one EC2 Instance type and Databricks requests EC2 instances via RunInstances API and not via EC2 Fleet. Is this feature not GA yet?

Re: Support for multiple EC2 instance types in a worker pool

Anonymous — Tue, 21 Mar 2023 06:52:53 GMT

Hi @Jagdeep Chahal

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: Support for multiple EC2 instance types in a worker pool

Debayan — Mon, 06 Mar 2023 05:05:49 GMT

Hi , this feature is still on private preview and one has to onboarded first to avail the same.

Re: Support for multiple EC2 instance types in a worker pool

abagshaw — Tue, 27 Jun 2023 18:37:39 GMT

Fleet instances on Databricks is now GA and available in all AWS workspaces - you can find more details here: https://docs.databricks.com/compute/aws-fleet-instances.html