Support Multiple Tasks DAG Inside a `for_each_task` Iteration

[NitinDatta]

Databricks Jobs `for_each_task` accepts exactly one nested task. For workloads that run the *same

multi-step pipeline over many datasets* — a very common metadata-driven pattern — this forces a

choice between two architectures, each of which gives up something important:

- **One task per iteration.** Iterations reuse warm compute within a run, so per-item compute

  acquisition cost is amortised. But because only one task is permitted, every step of the pipeline

  must be collapsed into a single notebook, and all per-step orchestration provided by the Jobs

  scheduler (per-step status, per-step retry, `condition_task` branching, repair-from-failed-step)

  is lost.

- **`run_job_task` per iteration.** Each item runs a child job with a proper multi-task graph, so

  all of the above is retained — but each item pays its own compute acquisition.

We are asking for the ability to define **multiple tasks within a single ForEach iteration**: a

small sub-DAG per item, with `depends_on`, `run_if`, and `condition_task` scoped to the iteration,

all tasks sharing that iteration's compute. This would remove the trade-off rather than making it

cheaper to live with.

[ThomazNeto · Accepted solution]

Hey droid,

First — your analysis is correct and still current. I re-checked the docs: a ForEach task takes exactly one nested task (you can't even nest another ForEach), and the dependency scope sits at the ForEach level, so you can't fan out a heterogeneous sub-graph inside the loop. You're not missing an option — the trade-off you laid out is real.

That said, before resigning yourself to the trade-off: the run_job_task-per-iteration path is cheaper than it used to be, and for a lot of metadata-driven fan-outs the compute-acquisition penalty is mostly solvable today.

- Serverless jobs compute for the child job. There's no cluster to acquire per item in the classic sense — startup is fast and you're not amortizing a warm cluster. For many fan-outs this alone neutralizes most of the "each item pays its own acquisition" cost. Benchmark it on your actual item profile rather than trusting a blanket claim, but it's usually the biggest lever.
- Instance pools if you're on classic compute. Pre-warmed nodes mean each child run grabs a ready node instead of provisioning cold — the classic amortization trick for many short child jobs.
- Concurrency. ForEach runs iterations concurrently (up to your configured concurrency), so wall-clock for run_job_task-per-item often isn't worse than serialized steps collapsed into one task.

Net: run_job_task per item keeps your real multi-task graph — per-step status, retry, condition_task, repair-from-failed-step — and with serverless or pools the cost gap narrows a lot. I'd measure that before collapsing anything.

The collapsed-single-notebook route can claw back some observability (sub-notebooks via dbutils.notebook.run, your own per-step try/except and logging), but as you already know it does NOT give you scheduler-native per-step status, repair-from-failed-step, or condition_task. Partial workaround, not a substitute — I wouldn't take it if per-step repair matters.

On the request itself: it's a good one, and worth filing properly — account team plus the Ideas/feature-request portal, not the forum (the forum won't route it to the Jobs PM reliably). One tip to make it land: frame it around what serverless does NOT solve, because that's the first thing Databricks will point to. Even with serverless, run_job_task-per-item still means N separate child runs — N run pages, no single iteration-scoped sub-DAG view, and clumsier task-value/context sharing within an item. A true in-iteration sub-DAG (depends_on / run_if / condition_task scoped to the iteration, sharing that iteration's context) is about orchestration fidelity and a unified run view, not just cost. Leading with that makes the ask much harder to wave off as "just use serverless."

[NitinDatta · Accepted solution]

Thanks Thomaz for detailed solutions and approaches.

On the serverless point we've benchmarked it, and it doesn't close the gap. Both figures below are serverless jobs compute on PERFORMANCE_OPTIMIZED, same 10-asset source system, same concurrency of 4. The only variable is architecture.
run_job_task per item (child job holding a 6-task graph): wave 1 averaged 3m43s; steady state settled at ~2m42s per item. Total 10 minutes wall clock.
ForEach with a single collapsed notebook task: wave 1 averaged 1m53s; steady state ~44s per item. Total 3.5 minutes wall clock.
That's 2. The per-item overhead in the fan-out is roughly two minutes, and it does not amortize — it's incurred per run, and each item is a run. Projected to our full 56-asset workload at the same concurrency: roughly 39 minutes versus 11.
In addition to that time depends on number tasks in for each dag more the tasks more is the time for each task.
We're not arguing cost alone — your framing advice is right, and we'll lead with orchestration fidelity and the unified iteration-scoped run view. But we'd ask that the cost dimension not be treated as solved by serverless. We measured it on the fastest tier available and it isn't. 

Regards

Nitin