Why is Databricks on AWS cluster start time less than 5 mins and EMR cluster start time is 15 mins?

[gud4eve]

We are migrating from AWS EMR to Databricks. One thing that we have noticed during the POCs is that Databricks cluster of same size and instance type takes much lesser time to start compared to EMR.

My understanding is Databricks also would be requesting instances from same AWS pool as EMR would do. Then why AWS's own service (EMR) is slow in getting the clusters up?

[-werners-]

Suppose the worker provisioning is identical between EMR and Databricks (I think they are the same, but am not certain), it is very possible that installing EMR on a cluster takes more time than installing Databricks. Databricks has worked hard to get their nodes up and running as fast as possible, perhaps Amazon did not do such a thing.

[gud4eve]

yes that's my assumption too. But do we have any documentation by Databricks stating that?

[Hubert-Dudek · Accepted solution]

I hear about trying to improve starting time at conferences for two years, so it is something like a never-ending story. Pools and serverless pools will offer further improvements. Recommended instance types are also usually better, as databricks is working with vendors on that. Additionally, I heard that GCC is now the fastest to start vms/cluster. For me, big improvements with deployment time would be that pools would have preinstalled libraries (instead of setting them on cluster level).

[karthik_p]

@gud4eve​ what kind of cluster you are using, have you configured pools. if not as @Werner Stinckens​ said there might be chance Databricks worked hard to get provisioning of instances in faster way

[gud4eve]

No we haven't configured pools