topic Re: Decrease frequency of Databricks Asset Bundle API in Data Engineering

Decrease frequency of Databricks Asset Bundle API

Neli — Fri, 26 Jul 2024 01:09:49 GMT

We are using DABs for our deployment and to invoke any workflow.

Behinds the scenes, it calls below API to get the status of workflow. Currently, it checks every few seconds. Is there a way to decrease this frequency from seconds to minutes.

GET /api/2.1/jobs/runs/get?

Why ?
Sometimes Job running in controlm fails with internal error due to network blip but workflow keeps executing.

Re: Decrease frequency of Databricks Asset Bundle API

Neli — Wed, 31 Jul 2024 13:06:25 GMT

Thank you @Retired_mod . Yes,custom polling mechanism is one option. Question for you, it means there is no way to override polling frequency of DAB API calls.

Behind the scenes,DAB calls below APIs :

GET /api/2.0/preview/scim/v2/Me
GET /api/2.0/workspace/get-status?path=/Workspace/datapipeline/deployments/bundles/prod/data-pipeline/state/terraform.tfstate
GET /api/2.0/workspace/export?direct_download=true&path=/Workspace/datapipeline/deployments/bundles/prod/data-pipeline/state/terraform.tfstate
POST /api/2.1/jobs/run-now
GET /api/2.1/jobs/runs/get?run_id=410841*******
GET /api/2.1/jobs/runs/get-output?run_id=410841******

Re: Decrease frequency of Databricks Asset Bundle API

Frank_Kennedy — Wed, 31 Jul 2024 14:19:44 GMT

Hey! If the API is checking the job status too frequently, you might want to consider implementing a custom polling mechanism. Instead of relying on the default frequency, you can build a simple script or function that pauses for a longer interval before checking the status again. Another approach could be to handle retries more gracefully, so a temporary network issue doesn’t cause the job to fail outright. Also, check if there are any API parameters or settings in Databricks that allow for adjusting the polling interval. Good luck, and hope you find a solution!

Re: Decrease frequency of Databricks Asset Bundle API

Neli — Wed, 31 Jul 2024 14:51:19 GMT

Sorry Frank, it comes as bundle. We dont have option to customise DAB. Definitely we can build custom script which can invoke same set of APIs with longer interval. Thank you for reviewing.