Idempotency can be ensured by providing the idempotency token. It's easy to pass the same through REST API as mentioned in the below doc:
https://kb.databricks.com/jobs/jobs-idempotency.html
The primary reason for multiple runs is the client submits the request and waits for the response from the server(Job Service). However, due to one or more reasons, the client does not get a response within its defined timeout period. After that, the client retries. However the initial request if successfully submitted on the job service will trigger the job run. The retry request will also trigger a job run causing duplicate job runs. Usage of idempotency token will ensure that the duplicate job runs are not triggered.
