Cluster Sizing

[Anonymous]

How big should my cluster be? How do I know how many nodes to use or the kind of instance to use?

[Anonymous · Accepted solution]

It depends a lot on the use case - ETL vs machine learning and training, etc. It also depends on the size of your data etc.

There's a good section on the documentation to give you some general guidance and process for sizing:

Best practices: Cluster configuration | Databricks on AWS

[User16857281974]

This article is based in part on the course produced by Databricks Academy called Optimizing Apache Spark on Databricks. These courses are 100% free, but also goes a bit deeper into the considerations required for making this decision, including usage, cloud costs and SLAs

[sajith_appukutt]

>How big should my cluster be? 

This would really depend on the use case. Some general guiding principles could be found here https://docs.databricks.com/clusters/cluster-config-best-practices.html