https://community.databricks.com/t5/data-engineering/databricks-vs-yarn-resource-utilization/m-p/18975#M12655 <P>This is not an Apple to Apple comparison. When you set 8GB as the executor memory in Yarn, then the container that is launched to run the executor JVM is getting 8GB of memory. Accordingly, the Xmx value of the heap is calculated. In Databricks, when you create a cluster with 8GB memory. The memory given to the executor JVM is less than that. This is because 8GB is the total instance memory.&nbsp;</P> Fri, 25 Jun 2021 19:06:46 GMT brickster_2018 2021-06-25T19:06:46Z https://community.databricks.com/t5/data-engineering/databricks-vs-yarn-resource-utilization/m-p/18974#M12654 <P>I have a spark-submit application that worked fine with 8GB executor memory in yarn. I am testing the same job against the Databricks cluster with the same executor memory. However, the jobs are running slower in Databricks.&nbsp;</P> Fri, 25 Jun 2021 19:06:06 GMT https://community.databricks.com/t5/data-engineering/databricks-vs-yarn-resource-utilization/m-p/18974#M12654 brickster_2018 2021-06-25T19:06:06Z https://community.databricks.com/t5/data-engineering/databricks-vs-yarn-resource-utilization/m-p/18975#M12655 <P>This is not an Apple to Apple comparison. When you set 8GB as the executor memory in Yarn, then the container that is launched to run the executor JVM is getting 8GB of memory. Accordingly, the Xmx value of the heap is calculated. In Databricks, when you create a cluster with 8GB memory. The memory given to the executor JVM is less than that. This is because 8GB is the total instance memory.&nbsp;</P> Fri, 25 Jun 2021 19:06:46 GMT https://community.databricks.com/t5/data-engineering/databricks-vs-yarn-resource-utilization/m-p/18975#M12655 brickster_2018 2021-06-25T19:06:46Z