Decrease frequency of Databricks Asset Bundle API
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-25-2024 06:09 PM
We are using DABs for our deployment and to invoke any workflow.
Behinds the scenes, it calls below API to get the status of workflow. Currently, it checks every few seconds. Is there a way to decrease this frequency from seconds to minutes.
GET /api/2.1/jobs/runs/get?
Why ?
Sometimes Job running in controlm fails with internal error due to network blip but workflow keeps executing.
- Labels:
-
Workflows
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-31-2024 06:03 AM - edited 07-31-2024 06:06 AM
Thank you @Retired_mod . Yes,custom polling mechanism is one option. Question for you, it means there is no way to override polling frequency of DAB API calls.
Behind the scenes,DAB calls below APIs :
GET /api/2.0/preview/scim/v2/Me
GET /api/2.0/workspace/get-status?path=/Workspace/datapipeline/deployments/bundles/prod/data-pipeline/state/terraform.tfstate
GET /api/2.0/workspace/export?direct_download=true&path=/Workspace/datapipeline/deployments/bundles/prod/data-pipeline/state/terraform.tfstate
POST /api/2.1/jobs/run-now
GET /api/2.1/jobs/runs/get?run_id=410841*******
GET /api/2.1/jobs/runs/get-output?run_id=410841******
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-31-2024 07:19 AM
Hey! If the API is checking the job status too frequently, you might want to consider implementing a custom polling mechanism. Instead of relying on the default frequency, you can build a simple script or function that pauses for a longer interval before checking the status again. Another approach could be to handle retries more gracefully, so a temporary network issue doesn’t cause the job to fail outright. Also, check if there are any API parameters or settings in Databricks that allow for adjusting the polling interval. Good luck, and hope you find a solution!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-31-2024 07:51 AM
Sorry Frank, it comes as bundle. We dont have option to customise DAB. Definitely we can build custom script which can invoke same set of APIs with longer interval. Thank you for reviewing.

