topic Re: Dynamic Allocation vs Cluster Auto-scaling in Warehousing & Analytics

Dynamic Allocation vs Cluster Auto-scaling

brickster_2018 — Fri, 21 Mar 2025 12:42:50 GMT

How is Cluster auto-scaling in Databricks different from Dynamic Allocation in Yarn

Re: Dynamic Allocation vs Cluster Auto-scaling

brickster_2018 — Thu, 24 Jun 2021 08:38:44 GMT

Dynamic allocation is a Spark feature exclusive for Yarn. Dynamic allocation looks for the idleness of the executor and is not shuffle aware. External shuffle service is mandatory to use Dynamic allocation because of this reason.

Databricks auto-scaling is shuffle aware and does not need external shuffle service. The algorithm used for the scale-up and scale-down is very much efficient. Also, the auto-scaling in Databricks provides configurations to the user to control the aggressiveness of scaling which is not available in Yarn.

Re: Dynamic Allocation vs Cluster Auto-scaling

shan_chandra — Wed, 03 Aug 2022 16:27:14 GMT

Additional reads on the same topic:

https://databricks.com/blog/2018/05/02/introducing-databricks-optimized-auto-scaling.html

https://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation

Re: Dynamic Allocation vs Cluster Auto-scaling

osr2020 — Fri, 17 Mar 2023 07:56:33 GMT

What does 'shuffle aware' mean in this context, and why does databricks not need an external shuffle service to scale up and down?

Re: Dynamic Allocation vs Cluster Auto-scaling

de-qrosh — Thu, 26 Mar 2026 19:42:52 GMT

From spark's documentation dynamic resource can be used on more then just YARN. Is it possible to use it in databricks as well and how does serverless work under the hood in these regards?