databricks list job runs api limit doesnt work

[aranjan99]

noticed that the limit parameter on the list job runs api: https://docs.databricks.com/api/workspace/jobs/listruns

doesnt work. no matter what limit i set, i always get all the runs which can be too many if i want all runs.
If i set offset as > 1, then limit is respected but then i can never get the run at offset 0. Can we address this.

[aranjan99]

same problem exists with this api too: https://docs.databricks.com/api/workspace/jobs/list

[aleksandra_ch]

Hi @aranjan99 ,

The limit parameter limits the number of runs per page, not the total cap. Make sure to have a single request with limit=N and do not follow page_token if the response includes one. That returns up to N most recent runs in a single page.

Hope it helps,

[aranjan99]

That is not correct. with limit=N and page_token = null. I still get a huge number of runs, way more than N. 
for eg this cli command: databricks jobs list-runs --job-id 716703755579168 --limit 10
returns 1000+ runs 

[aleksandra_ch]

@aranjan99 ,

With databricks jobs list-runs, pagination happens automatically. You can apply a filter on the result to keep only first N results:

databricks jobs list-runs --job-id "$JOB_ID" --limit 5 --output json | jq '.runs[:5]'

If you use REST API directly, you can explicitly choose not to check the next page. The below code will return exactly 5 runs:

curl -s -H "Authorization: Bearer $DATABRICKS_TOKEN" \
  --get "https://${DATABRICKS_INSTANCE}/api/2.1/jobs/runs/list" \
  --data-urlencode "job_id=${JOB_ID}" \
  --data-urlencode "limit=5" \

 Be aware that there is some work happening to add this explicit control to the Databricks CLI as well: https://github.com/databricks/cli/issues/1459#issuecomment-3774774740

Best regards,

[SteveOstrowski]

Hi @aranjan99,

To add some detail to the earlier reply: the confusion here comes from the fact that the REST API and the Databricks CLI handle the "limit" parameter differently.

REST API BEHAVIOR

In the REST API (POST or GET to /api/2.1/jobs/runs/list), the "limit" query parameter controls the page size, meaning how many runs are returned in a single response. The default is 25 and the maximum is 25. If there are more runs than the limit, the response includes a "next_page_token" field. If you simply do not follow that token, you get exactly N runs (or fewer if fewer exist). For example:

curl -s -H "Authorization: Bearer $DATABRICKS_TOKEN" \
"https://<your-instance>/api/2.1/jobs/runs/list?job_id=<JOB_ID>&limit=5"

That request returns at most 5 runs in a single response. If you stop there and do not paginate, you have your 5 most recent runs.

DATABRICKS CLI BEHAVIOR

The Databricks CLI auto-paginates by design. When you run:

databricks jobs list-runs --job-id 716703755579168 --limit 10

The CLI treats --limit as the page size for each underlying API call, but then automatically follows every next_page_token until all runs are returned. This is why you see 1000+ results even with --limit 10. The CLI is making multiple API calls behind the scenes (each fetching 10 runs per page) and concatenating all results.

This is a known behavior difference that is being tracked for improvement in the CLI:
https://github.com/databricks/cli/issues/1459

WORKAROUNDS

1. Use the REST API directly if you need precise control over the number of results. A single API call with limit=N and no pagination gives you exactly N runs.

2. If you prefer the CLI, pipe the output through jq to trim results:

databricks jobs list-runs --job-id <JOB_ID> --output json | jq '.runs[:10]'

3. If you use the Databricks Python SDK, the list_runs method also auto-paginates by default, but you can break out of the iterator after N results:

from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
runs = []
for run in w.jobs.list_runs(job_id=<JOB_ID>):
runs.append(run)
if len(runs) >= 10:
break

ABOUT THE OFFSET PARAMETER

You mentioned that setting offset > 1 causes limit to be respected. The "offset" parameter on the list runs API is deprecated in favor of token-based pagination (page_token). The interaction between offset and limit can produce unexpected results, so it is best to avoid offset entirely and use page_token-based pagination or the direct single-call approach described above.

DOCUMENTATION REFERENCES

- List Runs API reference: https://docs.databricks.com/api/workspace/jobs/listruns
- List Jobs API reference: https://docs.databricks.com/api/workspace/jobs/list
- Databricks CLI GitHub issue tracking this behavior: https://github.com/databricks/cli/issues/1459

* This reply used an agent system I built to research and draft this response based on the wide set of documentation I have available and previous memory. I personally review the draft for any obvious issues and for monitoring system reliability and update it when I detect any drift, but there is still a small chance that something is inaccurate, especially if you are experimenting with brand new features.