cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Example API call using 'has_more=true'

190809
Contributor

Can someone please provide an example while loop including has_more=true. I can't get pagination to work for the API endpoint '/jobs/runs/list/'. Thanks

2 REPLIES 2

Anonymous
Not applicable

@Rachel Cunningham​ :

Sure, here's an example of a while loop that utilizes pagination with the API endpoint /jobs/runs/list/:

import requests
 
url = "https://api.example.com/jobs/runs/list/"
headers = {
    "Authorization": "Bearer YOUR_ACCESS_TOKEN"
}
params = {
    "page_size": 10
}
 
has_more = True
while has_more:
    response = requests.get(url, headers=headers, params=params)
    response_json = response.json()
    data = response_json["data"]
    # Do something with the data
    
    has_more = response_json["has_more"]
    if has_more:
        params["after_id"] = response_json["after_id"]

In this example, we first define the API endpoint URL, the required authorization headers, and the initial query parameters. We then set has_more to True to start the loop.

Inside the loop, we send a GET request to the API endpoint with the defined headers and query parameters. We then extract the data from the JSON response and do something with it.

We then check if has_more is True in the JSON response. If it is, we update the after_id query parameter with the value from the JSON response and continue the loop. If it is False, we exit the loop.

Note that the page_size parameter is used to control the number of results per page. You may need to adjust this value depending on your API endpoint's pagination behavior.

arpit
Databricks Employee
Databricks Employee

Hi @Rachel Cunningham​ 

Could you please elaborate what you mean by "I can't get pagination to work"?

Is "has_more" set to "true" even when there are no more tasks to list? This is do you mean it doesn't list all runs or doesn't list tasks within each run? If the former, this is because the API is paginated. You will need to pass the "offset" parameter to list the next "page" of runs. If the latter, then you may need to pass the "expand_tasks" parameter. See the docs for more info

Let me know if you need more clarity on this.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group