topic Re: Get job run_id of run_job_task in an orchestration job in Data Engineering

Get job run_id of run_job_task in an orchestration job

MihaiTache — Fri, 18 Oct 2024 10:25:38 GMT

Hi,

I have a Databricks job which orchestrates the run of two jobs: job1 and job2 using run_job_task.

job2 depends on job1 and would need to use the run_id of job1 as a parameter. How can this be done?

I see that you can only easily access the task run id of the job {{tasks.job1.run_id}}

Re: Get job run_id of run_job_task in an orchestration job

filipniziol — Fri, 18 Oct 2024 11:47:58 GMT

Hi @MihaiTache ,

It seems you have found the answer. Do you need any other help?

Re: Get job run_id of run_job_task in an orchestration job

MihaiTache — Fri, 18 Oct 2024 12:03:35 GMT

{{tasks.job1.run_id}} provides the task run_id of the task under which job1 is run. What I'm looking for is the actual job run_id of job1, which doesn't seem available at first glance.

Re: Get job run_id of run_job_task in an orchestration job

filipniziol — Fri, 18 Oct 2024 12:31:28 GMT

Hi @MihaiTache ,

thank you for clarification.
As per documentation this value reference is not available out-of-the-box:
What is a dynamic value reference? | Databricks on AWS

A workaround is to use shared location available for 2 jobs.
As part of job1 you would store its value and job2 would read the saved value from shared location.

Re: Get job run_id of run_job_task in an orchestration job

Panda — Fri, 18 Oct 2024 14:57:48 GMT

@MihaiTache You can achieve this by utilizing a combination of dbutils.widgets.get and dbutils.jobs.taskValues.set. The approach involves extracting the run_id from Job1 and passing it as a value to Job2 using taskValues.set. This allows seamless communication between jobs within a workflow, enabling Job2 to access information or outputs from Job1 efficiently.

In Job1:

Extract necessary values (e.g., run_id) using dbutils.widgets.get to retrieve any required parameters passed to the job. Use dbutils.jobs.taskValues.set to store and make the value available for subsequent jobs.

In Job2:

Retrieve the value set by Job1 using dbutils.jobs.taskValues.get. This ensures Job2 starts with the required context, such as Job1’s run_id or other task-related values.

Example:

Job1 & Job 2 Setup screenshots: Review the attachment

Job1 Notebook:

job1_run_id = dbutils.widgets.get("job1_run_id")

dbutils.jobs.taskValues.set("job1_run_id", job1_run_id)

Job2 Notebook:

extract_job1_run_id = dbutils.widgets.get("extract_job1_run_id")

print(f"Received extract_job1_run_id: {extract_job1_run_id}")