topic Access to For Each run ids from jobs rest API in Data Engineering

Access to For Each run ids from jobs rest API

N_M — Tue, 24 Sep 2024 07:21:57 GMT

Hello Community

I'm using the for_each tasks in workflows, but I'm struggling to access to the job information through the job APIs.

In short, using the runs api (Get a single job run | Jobs API | REST API reference | Databricks on AWS), I'm able to access the single tasks run IDs of the specific run, when they are sub-jobs.

"tasks": [{'run_id': XXXXX,...},...]

In case of for each tasks, I can access the "root" task (name, run ID, everything), I can access the input parameters used in the sub jobs steps, but I can't access the run IDs of each iteration (but this information is easily accessible from the databricks portal).

On the other hand, if I get via API the info about such steps using the specific iteration run ID (that I take from the portal), the job is of course there, and it's also connected to the root ID via

{job_run_id:RUN_ID}

Am I missing something? How can I access the iteration run IDs knowing the run ID of the caller?

Re: Access to For Each run ids from jobs rest API

N_M — Tue, 24 Sep 2024 09:41:25 GMT

Small correction:
Iteration jobs can be connected to root job IDs via the attribute

{parent_run_id:RUN_ID}

Re: Access to For Each run ids from jobs rest API

N_M — Mon, 07 Oct 2024 09:58:42 GMT

I couldn't figure out anything to systematically access for_each tasks. I believe this is a quite important missing piece (unless of course I'm missing something).
Is there a way to ask to a developer or should I open a ticket through my company?

Re: Access to For Each run ids from jobs rest API

prabhatika — Mon, 18 Aug 2025 05:58:42 GMT

This feature would be extremely helpful in monitoring each task in the `foreachtask` task.