- 11369 Views
- 3 replies
- 1 kudos
Resolved! Dynamic Allocation vs Cluster Auto-scaling
How is Cluster auto-scaling in Databricks different from Dynamic Allocation in Yarn
- 11369 Views
- 3 replies
- 1 kudos
- 1 kudos
Additional reads on the same topic:https://databricks.com/blog/2018/05/02/introducing-databricks-optimized-auto-scaling.htmlhttps://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation
- 1 kudos
- 958 Views
- 1 replies
- 0 kudos
- 958 Views
- 1 replies
- 0 kudos
- 0 kudos
Run :-yarn cache clean
- 0 kudos
- 1472 Views
- 1 replies
- 0 kudos
Resolved! Unable to run 2 different applications with the same class name on a cluster
I have two jars with the same class name. It works fine on yarn. When trying to run these jars on the Databricks cluster, I run into issues. Why Databricks is having this limitation?
- 1472 Views
- 1 replies
- 0 kudos
- 0 kudos
When you run the jobs in Yarn, those are 2 different applications getting submitted on Yarn. Hence each application will have a separate Spark driver JVM's. In Databricks, a cluster has one JVM for the Spark driver. When applications with the same na...
- 0 kudos