- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-22-2023 01:56 PM
Tried using the databricks-sdk instead and just get an error on a different parameter -
"DatabricksError: Exactly 1 of virtual_cluster_size, num_workers or autoscale must be specified."
But, as you can see below, it definitely has "num_workers". And again, that JSON is literally pulled from the API docs....
```
cluster_spec={
"cluster_name": "single-node-cluster",
"node_type_id": "i3.xlarge",
"spark_version": "7.6.x-scala2.12",
"num_workers": 0,
"custom_tags": {
"ResourceClass": "SingleNode"
},
"spark_conf": {
"spark.databricks.cluster.profile": "singleNode",
"spark.master": "[*, 4]"
}
}
from databricks.sdk import WorkspaceClient
import json
dbrix_environment = 'production'
dbrix_host = f"https://XXXXXX-{dbrix_environment}.cloud.databricks.com"
dbrix_token = dbutils.secrets.get(scope = 'funstuff', key = 'choo-choo')
w = WorkspaceClient(
host = dbrix_host,
token = dbrix_token
)
w.clusters.create(json.dumps(cluster_spec))
```