topic Re: How to identify the mandatory fields of the create clusters API in Data Engineering

How to identify the mandatory fields of the create clusters API

guangyi — Fri, 30 Aug 2024 08:29:51 GMT

After several attempts I found some mandatory fields for cluster creation API: num_workers, spark_version, node_type_id. I’m not finding these fields directly against the API but via job cluster definition in the asset bundle yaml file.

I ask the ChatGPT the question and he gave almost the same answer and plus “cluster_name” field.

However, I cannot find the exact information on the API documentation either through “mandatory” or “required” keywords: https://docs.databricks.com/api/workspace/clusters/create.

Re: How to identify the mandatory fields of the create clusters API

guangyi — Fri, 30 Aug 2024 09:12:38 GMT

And also I found the `defaultValue` in the policy definition not working. Here I give the node_type_id allow list in the policy

"node_type_id": { "defaultValue": "Standard_D8s_v3", "type": "allowlist", "values": [ "Standard_D8s_v3", "Standard_D16s_v3" ] },

But when I inherit this policy in the asset bundle and try to create job, it still tell me the the validation failed:

job_clusters: - job_cluster_key: my_job_cluster new_cluster: policy_id: xxxxxxxxxxxxxxxxxxx

cannot create job: INVALID_PARAMETER_VALUE: Unknown node type id

Re: How to identify the mandatory fields of the create clusters API

VZLA — Fri, 03 Jan 2025 10:44:44 GMT

@guangyi thanks for your question!

I understand your concerns. Looking through the docs I could only find a few with the "required" metadata tag, while most seem to be implicitly assumed, e.g.: singleNode with num_workers 0, and similar requirements.

It'll be good if you can raise a Support Ticket, or provide this feedback or product improvement suggestion via the Databricks workspace portal.