cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Example API call using 'has_more=true'

190809
Contributor

Can someone please provide an example while loop including has_more=true. I can't get pagination to work for the API endpoint '/jobs/runs/list/'. Thanks

2 REPLIES 2

Anonymous
Not applicable

@Rachel Cunninghamโ€‹ :

Sure, here's an example of a while loop that utilizes pagination with the API endpoint /jobs/runs/list/:

import requests
 
url = "https://api.example.com/jobs/runs/list/"
headers = {
    "Authorization": "Bearer YOUR_ACCESS_TOKEN"
}
params = {
    "page_size": 10
}
 
has_more = True
while has_more:
    response = requests.get(url, headers=headers, params=params)
    response_json = response.json()
    data = response_json["data"]
    # Do something with the data
    
    has_more = response_json["has_more"]
    if has_more:
        params["after_id"] = response_json["after_id"]

In this example, we first define the API endpoint URL, the required authorization headers, and the initial query parameters. We then set has_more to True to start the loop.

Inside the loop, we send a GET request to the API endpoint with the defined headers and query parameters. We then extract the data from the JSON response and do something with it.

We then check if has_more is True in the JSON response. If it is, we update the after_id query parameter with the value from the JSON response and continue the loop. If it is False, we exit the loop.

Note that the page_size parameter is used to control the number of results per page. You may need to adjust this value depending on your API endpoint's pagination behavior.

arpit
Contributor III
Contributor III

Hi @Rachel Cunninghamโ€‹ 

Could you please elaborate what you mean by "I can't get pagination to work"?

Is "has_more" set to "true" even when there are no more tasks to list? This is do you mean it doesn't list all runs or doesn't list tasks within each run? If the former, this is because the API is paginated. You will need to pass the "offset" parameter to list the next "page" of runs. If the latter, then you may need to pass the "expand_tasks" parameter. See the docs for more info

Let me know if you need more clarity on this.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.