topic Cluster Sizing in Data Engineering

Cluster Sizing

Anonymous — Fri, 11 Jun 2021 04:11:01 GMT

How big should my cluster be? How do I know how many nodes to use or the kind of instance to use?

Re: Cluster Sizing

Anonymous — Fri, 11 Jun 2021 11:49:00 GMT

It depends a lot on the use case - ETL vs machine learning and training, etc. It also depends on the size of your data etc.

There's a good section on the documentation to give you some general guidance and process for sizing:

Best practices: Cluster configuration | Databricks on AWS

Re: Cluster Sizing

sajith_appukutt — Thu, 17 Jun 2021 23:58:06 GMT

>How big should my cluster be?

This would really depend on the use case. Some general guiding principles could be found here https://docs.databricks.com/clusters/cluster-config-best-practices.html

Re: Cluster Sizing

User16857281974 — Fri, 30 Jul 2021 22:17:05 GMT

This article is based in part on the course produced by Databricks Academy called Optimizing Apache Spark on Databricks. These courses are 100% free, but also goes a bit deeper into the considerations required for making this decision, including usage, cloud costs and SLAs