How to ensure that a Databricks Run Submit run invoked from Airflow only runs one time?

cgrant — Tue, 08 Jun 2021 22:35:20 GMT

I am running jobs on Databricks using the Run Submit API with Airflow. I have noticed that rarely, a particular run is run more than one time at once. Why?

Re: How to ensure that a Databricks Run Submit run invoked from Airflow only runs one time?

brickster_2018 — Wed, 23 Jun 2021 23:17:27 GMT

Idempotency can be ensured by providing the idempotency token. It's easy to pass the same through REST API as mentioned in the below doc:

https://kb.databricks.com/jobs/jobs-idempotency.html

The primary reason for multiple runs is the client submits the request and waits for the response from the server(Job Service). However, due to one or more reasons, the client does not get a response within its defined timeout period. After that, the client retries. However the initial request if successfully submitted on the job service will trigger the job run. The retry request will also trigger a job run causing duplicate job runs. Usage of idempotency token will ensure that the duplicate job runs are not triggered.

topic Re: How to ensure that a Databricks Run Submit run invoked from Airflow only runs one time? in Data Engineering

How to ensure that a Databricks Run Submit run invoked from Airflow only runs one time?

Re: How to ensure that a Databricks Run Submit run invoked from Airflow only runs one time?