Confusion about {{run_id}} and {{parent_run_id}} variables for Databricks jobs (Azure)

essentialDatabr — Thu, 23 Feb 2023 09:52:14 GMT

In Databricks jobs on Azure you can use the {{run_id}} and {{parent_run_id}}

variables for a specific run: https://docs.databricks.com/workflows/jobs/jobs.html

For Databricks jobs with only two or more tasks, then {{run_id}} seems to correspond to

task_run_id and {{parent_run_id}} seems to correspond to the job_run_id.

For Databricks jobs with only one task, then

{{parent_run_id}} seems to correspond to the task_run_id, but what does the

{{run_id}} correspond to? Is that the job_run_id?

Re: Confusion about {{run_id}} and {{parent_run_id}} variables for Databricks jobs (Azure)

Anonymous — Fri, 31 Mar 2023 15:44:41 GMT

@Kasper H :

Yes, you are correct in your understanding that in Databricks jobs with multiple tasks, the {{run_id}} variable corresponds to the task_run_id and the {{parent_run_id}} variable corresponds to the job_run_id.

For Databricks jobs with only one task, the {{parent_run_id}} variable still corresponds to the task_run_id, which is the unique identifier for that particular run of the task. In this case, the {{run_id}} variable would correspond to the job_run_id, which is the unique identifier for the entire job run that includes that single task.

So, in summary:

For Databricks jobs with multiple tasks: {{run_id}} = task_run_id, {{parent_run_id}} = job_run_id
For Databricks jobs with only one task: {{run_id}} = job_run_id, {{parent_run_id}} = task_run_id

topic Re: Confusion about {{run_id}} and {{parent_run_id}} variables for Databricks jobs (Azure) in Data Engineering

Confusion about {{run_id}} and {{parent_run_id}} variables for Databricks jobs (Azure)

Re: Confusion about {{run_id}} and {{parent_run_id}} variables for Databricks jobs (Azure)