Users often compare Databricks cluster vs Yarn Cluster. It's not an Apple to Apple comparison.
A Databricks cluster should be compared to a Spark Application that is submitted on Yarn. A Spark Application on Yarn will have a driver container and executor containers launched on the cluster nodes. The Application Master will run inside the Driver container (Yarn-Cluster mode).
A Databricks cluster also has a Driver container and the executor containers launched on the cluster nodes. Unlike Yarn, we launch only one executor per virtual machine. Application master in Yarn can be compared with the Chauffeur service in Databricks.
There are several benefits compared to Yarn in Databricks in this comparison:
- Support of multiple languages/sessions within the same cluster.
- Optimized and improved auto-scaling features. The auto-scaling algorithm used in Databricks is very much efficient than the Dynamic allocation feature in Yarn
- Faster and reliable with Spark's standalone scheduler.
