How to identify the mandatory fields of the create clusters API

guangyi
Contributor III

‎08-30-2024 01:29 AM

After several attempts I found some mandatory fields for cluster creation API: num_workers, spark_version, node_type_id. I’m not finding these fields directly against the API but via job cluster definition in the asset bundle yaml file.

I ask the ChatGPT the question and he gave almost the same answer and plus “cluster_name” field.

However, I cannot find the exact information on the API documentation either through “mandatory” or “required” keywords: https://docs.databricks.com/api/workspace/clusters/create.

0 Kudos

guangyi
Contributor III

‎08-30-2024 02:12 AM

And also I found the `defaultValue` in the policy definition not working. Here I give the node_type_id allow list in the policy 

    "node_type_id": {
        "defaultValue": "Standard_D8s_v3",
        "type": "allowlist",
        "values": [
            "Standard_D8s_v3",
            "Standard_D16s_v3"
        ]
    },

But when I inherit this policy in the asset bundle and try to create job, it still tell me the the validation failed:

job_clusters:
        - job_cluster_key: my_job_cluster
          new_cluster:
            policy_id: xxxxxxxxxxxxxxxxxxx

cannot create job: INVALID_PARAMETER_VALUE: Unknown node type id

0 Kudos

VZLA
Databricks Employee
Databricks Employee

‎01-03-2025 02:44 AM

@guangyi thanks for your question!

I understand your concerns. Looking through the docs I could only find a few with the "required" metadata tag, while most seem to be implicitly assumed, e.g.: singleNode with num_workers 0, and similar requirements.

It'll be good if you can raise a Support Ticket, or provide this feedback or product improvement suggestion via the Databricks workspace portal. 

0 Kudos