Databricks Vs Yarn - Resource Utilization

[brickster_2018]

I have a spark-submit application that worked fine with 8GB executor memory in yarn. I am testing the same job against the Databricks cluster with the same executor memory. However, the jobs are running slower in Databricks. 

[brickster_2018 · Accepted solution]

This is not an Apple to Apple comparison. When you set 8GB as the executor memory in Yarn, then the container that is launched to run the executor JVM is getting 8GB of memory. Accordingly, the Xmx value of the heap is calculated. In Databricks, when you create a cluster with 8GB memory. The memory given to the executor JVM is less than that. This is because 8GB is the total instance memory.