- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-10-2021 09:11 PM
How big should my cluster be? How do I know how many nodes to use or the kind of instance to use?
- Labels:
-
Cluster
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-11-2021 04:49 AM
It depends a lot on the use case - ETL vs machine learning and training, etc. It also depends on the size of your data etc.
There's a good section on the documentation to give you some general guidance and process for sizing:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-30-2021 03:17 PM
This article is based in part on the course produced by Databricks Academy called Optimizing Apache Spark on Databricks. These courses are 100% free, but also goes a bit deeper into the considerations required for making this decision, including usage, cloud costs and SLAs
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-17-2021 04:58 PM
>How big should my cluster be?
This would really depend on the use case. Some general guiding principles could be found here https://docs.databricks.com/clusters/cluster-config-best-practices.html
