we have noticed that more than 1000 jobs are in jobs list in shard . Due to which we are getting 'error_code': 'QUOTA_EXCEEDED', when submitting new jobs using job API

User16753724663
Databricks Employee
Databricks Employee
 

User16753724663
Databricks Employee
Databricks Employee

We can use the below api to list out the jobs and then use the delete job api:

https://docs.databricks.com/dev-tools/api/latest/jobs.html#list

List

Endpoint HTTP Method

2.0/jobs/list GET

Once we list out the jobs, then we can use below API to delete them:

https://docs.databricks.com/dev-tools/api/latest/jobs.html#delete

Delete

Endpoint HTTP Method

2.0/jobs/delete POST

Here is the Json that, we need to use:

{

"job_id": 1

}

Here is the sample python code for listing the job:

import requests
 
DOMAIN = '<your-domain>.cloud.databricks.com'
 
TOKEN = '<api-token>'
 
 
 
response = requests.get(
 
'https://%s/api/2.0/jobs/list' % (DOMAIN),
 
headers={'Authorization': 'Bearer %s' % TOKEN},
 
json={
 
}
 
)
 
 
 
if response.status_code == 200:
 
print(response.json())
 
else:
 
print("Error launching cluster: %s: %s" % (response.json()["error_code"], response.json()["message"]))

Here is the KB article as well on the same:

https://kb.databricks.com/jobs/howto-jobsdeleterestapi.html