Set Workflow Job Concurrency Limit

sage5616 — Wed, 15 Mar 2023 14:40:21 GMT

Hi Everyone,

I need a job to be triggered every 5 minutes. However, if that job is already running, it must not be triggered again until that run is finished. Hence, I need to set the maximum run concurrency for that job to only one instance at a time.

What needs to be configured to achieve that? Any recommendations and pointers would be much appreciated 😀

Best Regards

Re: Set Workflow Job Concurrency Limit

Anonymous — Fri, 17 Mar 2023 15:48:42 GMT

@Michael Okulik :

To ensure that a Databricks job is not triggered again until a running instance of the job is completed, you can set the maximum concurrency for the job to 1. Here's how you can configure this in Databricks:

Go to the Databricks workspace and navigate to the job you want to configure.
Click on the job name to open the job details page.
Click on the "Edit" button to edit the job configuration.
In the "Advanced" tab, scroll down to the "Concurrency" section.
Set the "Maximum number of runs" to 1.
Click on the "Save" button to save the job configuration.

With this configuration, if the job is already running, the next scheduled run will be skipped until the previous run is completed.

topic Set Workflow Job Concurrency Limit in Data Engineering

Set Workflow Job Concurrency Limit

Re: Set Workflow Job Concurrency Limit