How to limit max concurrent tasks runs in a job?

[yit337]

I read somewhere that there's a max_concurrent_task_runs property, but can't find it anywhere in the docs. So, how to limit the maximum concurrent tasks run in a job?

[amirabedhiafi]

Hello @yit337 !

I don't think there is a job level max_concurrent_task_runs setting for normal DAG tasks.

But there are 2 different concepts:

1. you can limit concurrent runs of the same job

resources:
  jobs:
    my_job:
      name: my_job
      max_concurrent_runs: 1

here you can limit how many runs of the same job can overlap. It does not limit how many tasks run in parallel inside one job run. https://docs.databricks.com/aws/en/jobs/configure-job

2. you can limit parallel iterations of a repeated task and use a For each task and set concurrency:

tasks:
  - task_key: process_items
    for_each_task:
      inputs: '["A", "B", "C", "D"]'
      concurrency: 2
      task:
        task_key: process_one_item
        notebook_task:
          notebook_path: ../src/process_one_item.py

 https://docs.databricks.com/aws/en/dev-tools/bundles/job-task-types

For normal separate tasks in the same job, concurrency is controlled by the DAG since tasks without dependencies can run in parallel and tasks with depends_on run after their dependencies. So to limit parallelism, you can group tasks into waves using dependencies. https://docs.databricks.com/aws/en/jobs/control-flow

[Ashwin_DSA · Accepted solution]

Hi @yit337,

There isn’t a max_concurrent_task_runs setting in Databricks Jobs. The only setting you get is max_concurrent_runs, which limits how many runs of the same job can be active at once, plus a workspace-wide limit of 2000 concurrent task runs. If you need to cap how many tasks from a single run execute in parallel, you currently have to do it yourself...either by structuring the DAG in waves (only N tasks can be runnable at a time) or by adding concurrency control inside the task code (for example, a thread pool with max_workers = N).

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.