ForEach Repair behaviour

[FelixBlom]

What is the behaviour of a 'repair' run on a ForEach task?

Will it only repair failed tasks & non-executed tasks (if cancelled early)? Or rerun all tasks? Where is this documented? Can this be added to the documentation of this task? https://docs.databricks.com/aws/en/jobs/tasks/for-each

[ThomazNeto]

Hi,

Fair question, and the honest answer is that the docs don't say. I went through the For each page, the repair page and the monitoring page and none of them mention how a repair treats iterations. So anything you read claiming "only failed iterations are re-run" is someone's observation, not documented behavior.

What is documented is the general rule, and it's worth reading with For each in mind. A repair run re-runs "only the subset of unsuccessful tasks and any dependent tasks", and the page is explicit that "A repair re-runs each unsuccessful task from the beginning. Lakeflow Jobs doesn't make tasks idempotent, so if a task wrote part of its output before it failed, re-running it can duplicate that data." The For each is a single task in the job graph; the iterations are runs of the nested task inside it, and the repair API only takes task keys (rerun_tasks, rerun_all_failed_tasks, rerun_dependent_tasks), not iteration ids. Reading the documented rule literally, if the For each task ended as failed or canceled, the repair re-runs that task from the beginning, which means all iterations. I would not bet production data on the opposite until Databricks writes it down.
https://docs.databricks.com/aws/en/jobs/repair-job-failures
https://docs.databricks.com/aws/en/jobs/tasks/for-each

Two practical things. First, it's a five-minute test: a For each over 5 values with a nested notebook that fails on one of them, then Repair run, and check the iterations table with the "Only failed iterations" filter off. Second, whatever the answer, make the nested task idempotent (MERGE, or check-and-skip on the input value) so a full re-run is harmless. That's what the docs are steering you towards anyway.
https://docs.databricks.com/aws/en/jobs/monitor#view-task-run-history-for-a-for-each-task

For the docs request, use the feedback control on that page; that goes straight to the docs team and they do pick these up. If you run the test, post the result here, I'd like to know too.

[FelixBlom · Accepted solution]

For completeness. I did the test. It's what I would've expected and is the most userfriendly task.

In short: Repairs do not re-run succeeded tasks. It only re-runs failed and scheduled tasks. I will ask for a doc change. Thanks for the pointers @ThomazNeto 

[aayush_410]

Behavior: A repair run on a job containing a For Each task follows the same rule as repair runs generally — it re-executes only the unsuccessful task runs (failed, or skipped/not-executed due to early cancellation or an upstream failure) and any tasks that depend on them. Successful task runs, and anything downstream of them, are left untouched and keep their original results.

For a For Each task specifically, this applies at the iteration level, not just at the parent task level. Each iteration of a For Each task is tracked as its own individual task run with its own success/failure/skip status — this is explicit in the Jobs API 2.2 iterations field (returned when run_id refers to a For Each task), where each object in that array has the identical schema to a normal task run object, including its own status. Since repair only re-runs unsuccessful runs and their dependents, and each iteration is its own run with independent status, a repair only re-executes the iterations that failed or never completed — successfully completed iterations are not rerun.

Where this is (and isn't) documented: The general repair-run behavior ("repair failed or canceled multi-task jobs by running only the subset of unsuccessful tasks and any dependent tasks... successful tasks... are not re-run") is documented on the Troubleshoot and repair job failures page. The per-iteration status tracking that this relies on for For Each tasks is documented separately, on the Jobs API 2.1→2.2 update reference (the iterations array). Neither page explicitly connects the two and states "repair on a For Each task only reruns failed/unexecuted iterations" as a single sentence — you're right that the For Each task page itself (the one you linked) doesn't mention repair behavior at all.

On getting it added: that's a reasonable doc gap to flag. Databricks docs pages have a feedback/"Was this page helpful?" mechanism at the bottom of each page — submitting through that on the For Each task page directly is the most likely way to get a docs engineer to add an explicit repair-run subsection there, since it ties two existing but disconnected doc sources together into the one place someone would actually look for it.