cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Decrease frequency of Databricks Asset Bundle API

Neli
New Contributor III

We are using DABs for our deployment and to invoke any workflow. 

Behinds the scenes, it calls below API to get the status of workflow. Currently, it checks every few seconds. Is there a way to decrease this frequency from seconds to minutes.  

GET /api/2.1/jobs/runs/get?

Why ?
Sometimes Job running in controlm fails with internal error due to network blip but workflow keeps executing. 

3 REPLIES 3

Neli
New Contributor III

Thank you @Retired_mod .  Yes,custom polling mechanism is one option. Question for you, it means there is no way to override polling frequency of DAB API calls. 

Behind the scenes,DAB calls below APIs :

GET /api/2.0/preview/scim/v2/Me 
GET /api/2.0/workspace/get-status?path=/Workspace/datapipeline/deployments/bundles/prod/data-pipeline/state/terraform.tfstate 
GET /api/2.0/workspace/export?direct_download=true&path=/Workspace/datapipeline/deployments/bundles/prod/data-pipeline/state/terraform.tfstate
POST /api/2.1/jobs/run-now 
GET /api/2.1/jobs/runs/get?run_id=410841*******
GET /api/2.1/jobs/runs/get-output?run_id=410841******

Frank_Kennedy
New Contributor II

Hey!  If the API is checking the job status too frequently, you might want to consider implementing a custom polling mechanism. Instead of relying on the default frequency, you can build a simple script or function that pauses for a longer interval before checking the status again. Another approach could be to handle retries more gracefully, so a temporary network issue doesnโ€™t cause the job to fail outright. Also, check if there are any API parameters or settings in Databricks that allow for adjusting the polling interval. Good luck, and hope you find a solution!

Neli
New Contributor III

Sorry Frank, it comes as bundle. We dont have option to customise DAB.  Definitely we can build custom script which can invoke same set of APIs with longer interval.  Thank you for reviewing. 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group