Hi @ChristianRRL  you're absolutely right, and I apologize for the earlier suggestion. I've verified that task values from child jobs are not propagated back through run_job tasks.

Your instinct about the REST API was correct. Here's the fix:

Solution: Add an intermediate notebook task in the orchestrator

Orchestrator:

  ├── Parent1 (run_job)

  ├── get_child_run_id (notebook task) ← NEW, depends on Parent1

  └── Parent2 (run_job, depends on get_child_run_id)

Notebook (`get_child_run_id`):

import requests

host = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiUrl().get()

token = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiToken().get()

headers = {"Authorization": f"Bearer {token}"}

# Get orchestrator run → find Parent1 → get child job run_id → find Child1

job_run_id = spark.conf.get("spark.databricks.job.runId")

orch_run = requests.get(f"{host}/api/2.1/jobs/runs/get",

    headers=headers, params={"run_id": job_run_id}).json()

parent1 = next(t for t in orch_run["tasks"] if t["task_key"] == "Parent1")

child_run = requests.get(f"{host}/api/2.1/jobs/runs/get-output",

    headers=headers, params={"run_id": parent1["run_id"]}).json()

child_job_run_id = child_run["metadata"]["run_id"]

child_job = requests.get(f"{host}/api/2.1/jobs/runs/get",

    headers=headers, params={"run_id": child_job_run_id}).json()

child1 = next(t for t in child_job["tasks"] if t["task_key"] == "Child1")

dbutils.jobs.taskValues.set(key="child1_run_id", value=str(child1["run_id"]))

Then in Parent 2, reference: {{tasks.get_child_run_id.values.child1_run_id}}

Can you check if this works?

Apologies again for the earlier response.

Regards,
Anuj