- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-09-2022 11:42 PM
We are migrating from AWS EMR to Databricks. One thing that we have noticed during the POCs is that Databricks cluster of same size and instance type takes much lesser time to start compared to EMR.
My understanding is Databricks also would be requesting instances from same AWS pool as EMR would do. Then why AWS's own service (EMR) is slow in getting the clusters up?
- Labels:
-
Emr
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-10-2022 04:26 AM
Suppose the worker provisioning is identical between EMR and Databricks (I think they are the same, but am not certain), it is very possible that installing EMR on a cluster takes more time than installing Databricks. Databricks has worked hard to get their nodes up and running as fast as possible, perhaps Amazon did not do such a thing.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-10-2022 04:55 AM
yes that's my assumption too. But do we have any documentation by Databricks stating that?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2022 09:22 AM
I hear about trying to improve starting time at conferences for two years, so it is something like a never-ending story. Pools and serverless pools will offer further improvements. Recommended instance types are also usually better, as databricks is working with vendors on that. Additionally, I heard that GCC is now the fastest to start vms/cluster. For me, big improvements with deployment time would be that pools would have preinstalled libraries (instead of setting them on cluster level).
My blog: https://databrickster.medium.com/
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-13-2022 07:41 AM
@gud4eve what kind of cluster you are using, have you configured pools. if not as @Werner Stinckens said there might be chance Databricks worked hard to get provisioning of instances in faster way
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-18-2022 12:22 AM
No we haven't configured pools
