Missing Fields in Databricks REST API Documentation & SDK Due to OpenAPI Spec Gaps

costi9992 — Wed, 25 Sep 2024 18:40:12 GMT

Hi Community,

I've been working with the Databricks REST APIs and noticed some inconsistencies between the API documentation and the actual API responses. Specifically, there are a few fields returned in the API responses that are not documented but are present in the API response samples.

The missing fields are as follows:

For /api/2.1/clusters/list https://docs.databricks.com/api/workspace/clusters/list:
- `disk_spec`
- `last_activity_time`

For /api/2.1/clusters/get https://docs.databricks.com/api/workspace/clusters/get:
- `disk_spec`
- `last_activity_time`

For /api/2.1/clusters/list-node-types https://docs.databricks.com/api/workspace/clusters/listnodetypes:
- `node_instance_type.instance_family`

For /api/2.0/pipelines https://docs.databricks.com/api/workspace/pipelines/listpipelines:
- `prev_page_token`

These fields are crucial for certain use cases, and it would be helpful to have them officially documented in the API documentation.

Moreover, this issue also impacts the **Databricks SDK**, as the SDK classes are generated from OpenAPI specs using the **Databricks SDK Generator**. Since these fields are missing from the OpenAPI spec, we cannot access them when using the SDK, which limits functionality in certain scenarios.

Has anyone else encountered this issue? Is there a plan to update the documentation and the OpenAPI specs to include these fields so they can be used in both the API and SDK?

Thanks in advance for your help!

Re: Missing Fields in Databricks REST API Documentation & SDK Due to OpenAPI Spec Gaps

Walter_C — Fri, 06 Dec 2024 11:53:40 GMT

Hello thanks for your question, in regards the last_time_activity and disk_spec this fields have been deprecated and this is the reason why it is no longer showing in the API docs, you can refer to https://kb.databricks.com/clusters/databricks-api-last_activity_time-attribute-shows-incorrect-timestamp/version/1?kb_language=en_US

topic Re: Missing Fields in Databricks REST API Documentation & SDK Due to OpenAPI Spec Gaps in Data Engineering

Missing Fields in Databricks REST API Documentation & SDK Due to OpenAPI Spec Gaps

Re: Missing Fields in Databricks REST API Documentation & SDK Due to OpenAPI Spec Gaps