Hi community,
We have setup a Databricks cluster as Single node with num_workers=0 . Sometimes the cluster automatically resizes to e.g. 10 workers. When I edit the cluster subsequently it gives an error that num_workers is not allowed for Single node clusters and if I delete the 10 workers, the setting and error will be gone. This looks like a bug? What could cause this odd behavior?
Our cluster is defined as follows:
{
"num_workers": 0,
"cluster_name": "SingleNode-test",
"spark_version": "11.3.x-scala2.12",
"spark_conf": {
"spark.master": "local[*]",
"spark.databricks.cluster.profile": "singleNode",
"spark.databricks.delta.preview.enabled": "true"
},
"azure_attributes": {
"first_on_demand": 1,
"availability": "ON_DEMAND_AZURE",
"spot_bid_max_price": -1
},
"node_type_id": "Standard_DS3_v2",
"driver_node_type_id": "Standard_DS3_v2",
"ssh_public_keys": [],
"custom_tags": {
"ResourceClass": "SingleNode"
},
"spark_env_vars": {},
"autotermination_minutes": 90,
"enable_elastic_disk": true,
"cluster_source": "UI",
"init_scripts": [],
"enable_local_disk_encryption": false,
"cluster_id": "<REMOVED>"
}