Changes in Databricks rest api

Databrixer — Fri, 13 Sep 2024 10:00:49 GMT

Hi all,
Just wanted to ask if there is an announcement regarding changes on Databricks REST API?
Because I had weird experiences in using the REST API.
Specifically for Query History API

Last Aug 29, 2024.
I created a script to pull get request in the endpoint. I saw a key in the response named:

'query_source': {'driver_info': {'driver_name': '', 'version_number': '2.0', etc..'},

I can get all the response but not with that specific key. (use case is to determine where it is used ex:NB, Genie space, etc.)
I checked the query params, but there is nothing related to query_source. I even also enabled metrics, still no luck.

But then with my playful mind, I added this on my parameters:

"include_query_source": 'true', - which is not part of the documentation - and you guess it, it worked.

Now as I am writing this, the 'query_source' is now missing in the response sample in the documentation. But still works.
It is better if we have and announcement regarding the changes of the endpoints because this might cause issues in the future when we decide to create a workflow for this kind of use case.

Re: Changes in Databricks rest api

szymon_dybczak — Fri, 13 Sep 2024 11:10:45 GMT

Yeah, totally agree with you. It should be documented/mentioned somewhere. Or maybe the API should be versioned, so if they introduce a new version, it won't break existing workflows.
Anyway, thanks for sharing!

Re: Changes in Databricks rest api

Databrixer — Fri, 13 Sep 2024 11:23:27 GMT

Agree. They have announcements on other endpoints in the documentation but it is better to have a dedicated page for the list of versions.

topic Changes in Databricks rest api in Administration & Architecture

Changes in Databricks rest api

Re: Changes in Databricks rest api

Re: Changes in Databricks rest api