Hello! You can achieve this in two ways:

Option 1: Orchestrator Job with Task Values
Reference: https://learn.microsoft.com/en-us/azure/databricks/jobs/task-values

Create three separate jobs and one orchestrator job that calls the others. In each transformation, set a variable called job_id and use:

python
dbutils.jobs.taskValues.set(key="first_job_id", value=job_id)

This creates a variable `first_job_id` that can be accessed in other tasks using:

{{tasks.task_for_job_1.values.first_job_id}}

Option 2: Built-in Job Parameters
Reference: https://docs.databricks.com/aws/en/jobs/dynamic-value-references

You can simply use Databricks' built-in dynamic references in your job parameters:

json
{
"parameters": [
{
"name": "my_job_id",
"default": "{{job.id}}"
},
{
"name": "run_date",
"default": "{{job.start_time.iso_date}}"
}
]
}