cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Access to For Each run ids from jobs rest API

N_M
Contributor

Hello Community

I'm using the for_each tasks in workflows, but I'm struggling to access to the job information through the job APIs.

In short, using the runs api (Get a single job run | Jobs API | REST API reference | Databricks on AWS), I'm able to access the single tasks run IDs of the specific run, when they are sub-jobs.

"tasks": [{'run_id': XXXXX,...},...]

 In case of for each tasks, I can access the "root" task (name, run ID, everything), I can access the input parameters used in the sub jobs steps, but I can't access the run IDs of each iteration (but this information is easily accessible from the databricks portal).

On the other hand, if I get via API the info about such steps using the specific iteration run ID (that I take from the portal), the job is of course there, and it's also connected to the root ID via

{job_run_id:RUN_ID}

  Am I missing something? How can I access the iteration run IDs knowing the run ID of the caller?

2 REPLIES 2

N_M
Contributor

Small correction:
Iteration jobs can be connected to root job IDs via the attribute

{parent_run_id:RUN_ID}

 

N_M
Contributor

I couldn't figure out anything to systematically access for_each tasks. I believe this is a quite important missing piece (unless of course I'm missing something).
Is there a way to ask to a developer or should I open a ticket through my company?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group