cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Why does my Single Node cluster automatically resize num_workers?

JonD
New Contributor III

Hi community,

We have setup a Databricks cluster as Single node with num_workers=0 . Sometimes the cluster automatically resizes to e.g. 10 workers. When I edit the cluster subsequently it gives an error that num_workers is not allowed for Single node clusters and if I delete the 10 workers, the setting and error will be gone. This looks like a bug? What could cause this odd behavior?

Our cluster is defined as follows:

{
    "num_workers": 0,
    "cluster_name": "SingleNode-test",
    "spark_version": "11.3.x-scala2.12",
    "spark_conf": {
        "spark.master": "local[*]",
        "spark.databricks.cluster.profile": "singleNode",
        "spark.databricks.delta.preview.enabled": "true"
    },
    "azure_attributes": {
        "first_on_demand": 1,
        "availability": "ON_DEMAND_AZURE",
        "spot_bid_max_price": -1
    },
    "node_type_id": "Standard_DS3_v2",
    "driver_node_type_id": "Standard_DS3_v2",
    "ssh_public_keys": [],
    "custom_tags": {
        "ResourceClass": "SingleNode"
    },
    "spark_env_vars": {},
    "autotermination_minutes": 90,
    "enable_elastic_disk": true,
    "cluster_source": "UI",
    "init_scripts": [],
    "enable_local_disk_encryption": false,
    "cluster_id": "<REMOVED>"
}

1 ACCEPTED SOLUTION

Accepted Solutions

JonD
New Contributor III

I think the issue is solved, at least it didn't occur in the last month. We monitored this via Azure Log Analytics. Maybe it was solved due to some patch/update, thanks anyway!

View solution in original post

3 REPLIES 3

Anonymous
Not applicable

@Jon Daal​ : The behavior you are experiencing where the cluster is automatically resizing to add worker nodes, even though it is defined as a Single Node cluster with "num_workers": 0, is unexpected and may be a bug in Databricks.

To investigate the issue further, you can try checking the logs for the cluster and see if there are any errors or warnings related to the automatic resizing.

Anonymous
Not applicable

Hi @Jon Daal​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!

JonD
New Contributor III

I think the issue is solved, at least it didn't occur in the last month. We monitored this via Azure Log Analytics. Maybe it was solved due to some patch/update, thanks anyway!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group