we have noticed that more than 1000 jobs are in jobs list in shard . Due to which we are getting 'error_code': 'QUOTA_EXCEEDED', when submitting new jobs using job API
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-16-2021 10:25 AM
1 REPLY 1
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-16-2021 10:26 AM
We can use the below api to list out the jobs and then use the delete job api:
https://docs.databricks.com/dev-tools/api/latest/jobs.html#list
List
Endpoint HTTP Method
2.0/jobs/list GET
Once we list out the jobs, then we can use below API to delete them:
https://docs.databricks.com/dev-tools/api/latest/jobs.html#delete
Delete
Endpoint HTTP Method
2.0/jobs/delete POST
Here is the Json that, we need to use:
{
"job_id": 1
}
Here is the sample python code for listing the job:
import requests
DOMAIN = '<your-domain>.cloud.databricks.com'
TOKEN = '<api-token>'
response = requests.get(
'https://%s/api/2.0/jobs/list' % (DOMAIN),
headers={'Authorization': 'Bearer %s' % TOKEN},
json={
}
)
if response.status_code == 200:
print(response.json())
else:
print("Error launching cluster: %s: %s" % (response.json()["error_code"], response.json()["message"]))
Here is the KB article as well on the same: