cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to get the list of all jobs available for a particular user?

Maverick1
Valued Contributor II

As of now, if I try to list the jobs via "list job" API then there is a limit of 25 jobs only.

Is there a way to list all the available/visible jobs to a user?

1 ACCEPTED SOLUTION

Accepted Solutions

Maverick1
Valued Contributor II

@Kaniz Fatmaโ€‹  @Arvind Ravishโ€‹ :

Yes the user can generate the API token but the limit is on the "get jobs" API level in version 2.1.

One workaround that I found is to create a while loop function and pass the offset param variable to "list job" API with value as 25.

Then reiterate and check the json response at every iteration. If the response have a key value pair as {"has_more: False"} then break the loop. Also, at every iteration, keep on appending the job_name or job_id or any other relevant info. from json response in a list.

Finally you will be having a list of all jobs that is visible to the current user.

Alternatively you can use "list job" API version 2.0. This version doesn't have the limit of 25 but it won't return the multi-tasks inside the job and has a hard limit of max 2000 jobs.

View solution in original post

4 REPLIES 4

User16764241763
Honored Contributor

Hello @Saurabh Vermaโ€‹ 

Can the user generate the API token in the workspace and try to use the API?

Kaniz_Fatma
Community Manager
Community Manager

Hi @Saurabh Vermaโ€‹, We havenโ€™t heard from you on the last response from @Arvind Ravishโ€‹, and I was checking back to see if his suggestions helped you. Or else, If you have any solution, please share it with the community as it can be helpful to others.

Maverick1
Valued Contributor II

@Kaniz Fatmaโ€‹  @Arvind Ravishโ€‹ :

Yes the user can generate the API token but the limit is on the "get jobs" API level in version 2.1.

One workaround that I found is to create a while loop function and pass the offset param variable to "list job" API with value as 25.

Then reiterate and check the json response at every iteration. If the response have a key value pair as {"has_more: False"} then break the loop. Also, at every iteration, keep on appending the job_name or job_id or any other relevant info. from json response in a list.

Finally you will be having a list of all jobs that is visible to the current user.

Alternatively you can use "list job" API version 2.0. This version doesn't have the limit of 25 but it won't return the multi-tasks inside the job and has a hard limit of max 2000 jobs.

Awesome @Saurabh Vermaโ€‹!

Thank you for sharing the great workaround and choosing the best answer.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group