This article is based in part on the course produced by Databricks Academy called Optimizing Apache Spark on Databricks. These courses are 100% free, but also goes a bit deeper into the considerations required for making this decision, including usage, cloud costs and SLAs