topic Re: API Call to return more than 100 jobs in Data Engineering

API Call to return more than 100 jobs

dpc — Wed, 26 Nov 2025 15:42:41 GMT

Hello

I have around 150 jobs and this is likely to increase.

I use this call to get all the jobs and write them into a list called json.

My logic here is to match a name to a job id and run the job using the job id.

response = requests.get(hostHTTPS, json = additional_parameters, headers=auth).json()

HostHTTPS is basically this:

hostHTTPS = ''.join([databricksHost, "/api/2.1/jobs/list?limit=100"])

That goes off and returns 100 jobs - all good

If I change limit to more than 100, I get (e.g. 200):

{'error_code': 'INVALID_PARAMETER_VALUE', 'message': 'Invalid limit 200 - it has to be no more than 100.', 'details': [{'@type': 'type.googleapis.com/google.rpc.RequestInfo'

How do I increase this as 100 is a pretty small number?

Thanks

Re: API Call to return more than 100 jobs

bianca_unifeye — Wed, 26 Nov 2025 15:53:50 GMT

Please refer to https://docs.databricks.com/api/workspace/jobs/list?

The official documentation for the Jobs API (GET /api/2.2/jobs/list) states that the limit parameter controls how many jobs are returned per call, and the API enforces a cap (100).

Because you have ~150 jobs and may have more later, you should loop over subsequent pages until you've retrieved all jobs. The API supports pagination: the response will include a next_page_token (or an equivalent mechanism) when there are more jobs to fetch.

https://docs.databricks.com/aws/en/reference/jobs-api-2-2-updates

Re: API Call to return more than 100 jobs

dpc — Wed, 26 Nov 2025 16:33:30 GMT

Thanks. I will try that

Re: API Call to return more than 100 jobs

dpc — Wed, 03 Dec 2025 08:48:20 GMT

Looping using next_page_token works well, thanks @bianca_unifeye