- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-04-2026 11:51 PM
I read somewhere that there's a max_concurrent_task_runs property, but can't find it anywhere in the docs. So, how to limit the maximum concurrent tasks run in a job?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-05-2026 03:04 AM
Hello @yit337 !
I don't think there is a job level max_concurrent_task_runs setting for normal DAG tasks.
But there are 2 different concepts:
1. you can limit concurrent runs of the same job
resources:
jobs:
my_job:
name: my_job
max_concurrent_runs: 1here you can limit how many runs of the same job can overlap. It does not limit how many tasks run in parallel inside one job run. https://docs.databricks.com/aws/en/jobs/configure-job
2. you can limit parallel iterations of a repeated task and use a For each task and set concurrency:
tasks:
- task_key: process_items
for_each_task:
inputs: '["A", "B", "C", "D"]'
concurrency: 2
task:
task_key: process_one_item
notebook_task:
notebook_path: ../src/process_one_item.pyhttps://docs.databricks.com/aws/en/dev-tools/bundles/job-task-types
For normal separate tasks in the same job, concurrency is controlled by the DAG since tasks without dependencies can run in parallel and tasks with depends_on run after their dependencies. So to limit parallelism, you can group tasks into waves using dependencies. https://docs.databricks.com/aws/en/jobs/control-flow
Senior BI/Data Engineer | Microsoft MVP Data Platform | Microsoft MVP Power BI | Power BI Super User | C# Corner MVP
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-05-2026 05:10 AM
Hi @yit337,
There isn’t a max_concurrent_task_runs setting in Databricks Jobs. The only setting you get is max_concurrent_runs, which limits how many runs of the same job can be active at once, plus a workspace-wide limit of 2000 concurrent task runs. If you need to cap how many tasks from a single run execute in parallel, you currently have to do it yourself...either by structuring the DAG in waves (only N tasks can be runnable at a time) or by adding concurrency control inside the task code (for example, a thread pool with max_workers = N).
If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.
Ashwin | Delivery Solution Architect @ Databricks
Helping you build and scale the Data Intelligence Platform.
***Opinions are my own***