topic What is the best expected starting time for a cluster when using a pool? in Data Engineering

What is the best expected starting time for a cluster when using a pool?

PaulP — Fri, 14 Oct 2022 19:36:54 GMT

Hi!

I'm doing some tests to get an idea of how much time could be saved starting a cluster by using a pool and was wondering if the results I get are what should be expected.

We're using AWS Databricks and used i3.xlarge as instance type (if that matters).

When not using pool it takes about 3 minutes for my cluster to start. When using the pool it takes about a minute which still seems a long time to me since there is an instance up and running.

Is it what can reasonably be expected or is there some way to improve the cluster creation speed?

Re: What is the best expected starting time for a cluster when using a pool?

karthik_p — Mon, 17 Oct 2022 16:22:21 GMT

@Paul Pelletier usually pools are 4 times faster, pools are best way to speed up cluster creation as they overcome DBR, and Demon services are alredy been fetched. cluster auto scaling should be also fine, but pools are recommended as far as i know

Re: What is the best expected starting time for a cluster when using a pool?

Hubert-Dudek — Thu, 20 Oct 2022 13:20:49 GMT

One minute is pretty good. There are also differences between instance types (some are faster to start).

Re: What is the best expected starting time for a cluster when using a pool?

Anonymous — Sat, 26 Nov 2022 07:11:49 GMT

Hi @Paul Pelletier

Hope all is well!

Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!