The reason Spark is so much faster on Databricks is because it is a managed service.  We have full control over the integration along storage, memory management, and networking to name a few.  Spark running on Databricks is anywhere between 5 - 100x faster depending on the workload.