Showing results for 
Search instead for 
Did you mean: 
Community Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
Showing results for 
Search instead for 
Did you mean: 

How the Scale up process done in the databricks cluster?

New Contributor III

For my AWS databricks cluster, i configured shared computer with 1min worker node and 3 max worker node, initailly only one worker node and driver node instance is created in the AWS console. 
Is there any rule set by databricks for scale up the next node like any threshold exceeds in the initial node(min node)?

How the scale up process done from one node to another node by databricks automatically?




Community Manager
Community Manager

Hi @Nandhini_Kumar

  1. Cluster Configuration:

    • When you create a Databricks cluster, you have several options for compute configuration. These choices impact performance, cost, and scalability.
    • Two primary types of computing are available:
      • All-purpose compute: Shared by multiple users, suitable for ad-hoc analysis, data exploration, or development.
      • Job compute: Used for operationalizing code after development. Job computes terminate when the job ends, reducing resource usage and cost.
    • You can also choose between single-node (for small workloads) and multi-node (for larger, distribute...1.
  2. Autoscaling:

    • Databricks provides an optimized autoscaling service that dynamically adjusts the number of workers based on load.
    • Here’s how it works:
      • When you provide a range for the number of workers, Databricks selects the appropriate number of workers needed for your job.
      • Autoscaling ensures efficient resource utilization without manual intervention.
      • Under low utilization, clusters can be scaled down aggressively while maintaining responsiveness.
      • Autoscaling helps balance cost and performance2.
  3. Thresholds and Scaling Rules:

  4. Visibility and Control:

In summary, Databricks optimizes cluster scaling by dynamically reallocating workers, ensuring efficient resource utilization, and maintaining responsiveness. While there isn’t a fixed rule for scaling thresholds, autoscaling adapts to workload demands automatically. 🚀🔍

For more detailed configuration options, you can refer to the official Databricks documentation1.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!