https://community.databricks.com/t5/data-engineering/why-is-databricks-on-aws-cluster-start-time-less-than-5-mins-and/m-p/28256#M20079 <P>yes that's my assumption too. But do we have any documentation by Databricks stating that?</P> Mon, 10 Oct 2022 11:55:36 GMT gud4eve 2022-10-10T11:55:36Z https://community.databricks.com/t5/data-engineering/why-is-databricks-on-aws-cluster-start-time-less-than-5-mins-and/m-p/28254#M20077 <P>We are migrating from AWS EMR to Databricks. One thing that we have noticed during the POCs is that Databricks cluster of same size and instance type takes much lesser time to start compared to EMR.</P><P>My understanding is Databricks also would be requesting instances from same AWS pool as EMR would do. Then why AWS's own service (EMR) is slow in getting the clusters up?</P> Mon, 10 Oct 2022 06:42:57 GMT https://community.databricks.com/t5/data-engineering/why-is-databricks-on-aws-cluster-start-time-less-than-5-mins-and/m-p/28254#M20077 gud4eve 2022-10-10T06:42:57Z https://community.databricks.com/t5/data-engineering/why-is-databricks-on-aws-cluster-start-time-less-than-5-mins-and/m-p/28255#M20078 <P>Suppose the worker provisioning is identical between EMR and Databricks (I think they are the same, but am not certain), it is very possible that installing EMR on a cluster takes more time than installing Databricks. Databricks has worked hard to get their nodes up and running as fast as possible, perhaps Amazon did not do such a thing.</P> Mon, 10 Oct 2022 11:26:30 GMT https://community.databricks.com/t5/data-engineering/why-is-databricks-on-aws-cluster-start-time-less-than-5-mins-and/m-p/28255#M20078 -werners- 2022-10-10T11:26:30Z https://community.databricks.com/t5/data-engineering/why-is-databricks-on-aws-cluster-start-time-less-than-5-mins-and/m-p/28256#M20079 <P>yes that's my assumption too. But do we have any documentation by Databricks stating that?</P> Mon, 10 Oct 2022 11:55:36 GMT https://community.databricks.com/t5/data-engineering/why-is-databricks-on-aws-cluster-start-time-less-than-5-mins-and/m-p/28256#M20079 gud4eve 2022-10-10T11:55:36Z https://community.databricks.com/t5/data-engineering/why-is-databricks-on-aws-cluster-start-time-less-than-5-mins-and/m-p/28257#M20080 <P>@gud4eve​&nbsp;what kind of cluster you are using, have you configured pools. if not as @Werner Stinckens​&nbsp;said there might be chance Databricks worked hard to get provisioning of instances in faster way </P> Thu, 13 Oct 2022 14:41:18 GMT https://community.databricks.com/t5/data-engineering/why-is-databricks-on-aws-cluster-start-time-less-than-5-mins-and/m-p/28257#M20080 karthik_p 2022-10-13T14:41:18Z https://community.databricks.com/t5/data-engineering/why-is-databricks-on-aws-cluster-start-time-less-than-5-mins-and/m-p/28258#M20081 <P>I hear about trying to improve starting time at conferences for two years, so it is something like a never-ending story. Pools and serverless pools will offer further improvements. Recommended instance types are also usually better, as databricks is working with vendors on that. Additionally, I heard that GCC is now the fastest to start vms/cluster. For me, big improvements with deployment time would be that pools would have preinstalled libraries (instead of setting them on cluster level).</P> Sun, 16 Oct 2022 16:22:33 GMT https://community.databricks.com/t5/data-engineering/why-is-databricks-on-aws-cluster-start-time-less-than-5-mins-and/m-p/28258#M20081 Hubert-Dudek 2022-10-16T16:22:33Z https://community.databricks.com/t5/data-engineering/why-is-databricks-on-aws-cluster-start-time-less-than-5-mins-and/m-p/28259#M20082 <P>No we haven't configured pools</P> Tue, 18 Oct 2022 07:22:50 GMT https://community.databricks.com/t5/data-engineering/why-is-databricks-on-aws-cluster-start-time-less-than-5-mins-and/m-p/28259#M20082 gud4eve 2022-10-18T07:22:50Z