topic Re: Uncertainty on Databricks job variables: {{run_id}}, {{parent_run_id}}. in Data Engineering

Uncertainty on Databricks job variables: {{run_id}}, {{parent_run_id}}.

AndrewSilver — Fri, 17 Nov 2023 13:06:38 GMT

In Azure's Databricks jobs, {{run_id}} and {{parent_run_id}} serve as variables. In jobs with multiple tasks, {{run_id}} aligns with task_run_id, while {{parent_run_id}} matches job_run_id. In single-task jobs, {{parent_run_id}} aligns with task_run_id, yet {{run_id}}'s specific correlation remains unclear—potentially job_run_id.

Re: Uncertainty on Databricks job variables: {{run_id}}, {{parent_run_id}}.

kulkpd — Fri, 17 Nov 2023 17:40:23 GMT

I am using job with single task and multiple retry.
Upon job retry the run_id get changed, I tried to using {{parent_run_id}} but never worked so switched to
val parentRunId = dbutils.notebook.getContext.tags("jobRunOriginalAttempt")