Allowing a job parameter that is pushed down to be overridden

[dpc]

Hello

I have a Job that calls a job which calls a job (this could go on)

I want to generate an id for each job and log that id along with a parent id and job name

So, I am creating an id at each level as the first task

Then passing this to the next level as the ParentId

At the next level it then generates and id and would then log the parent (pushed) and id

So (e.g):

Name	ParentId	Id
Top Level	0	123
Level 2	123	456
Level 3	456	789

The top 2 levels are fine but when I get to the 3rd (and beyond),

I cannot pass the Level 2 Id as the parent because it's already defined and will not let me override

Is this possible?

I can do this in Azure Data Factory but want to replace the functionality

Thanks

[jooguilhermesc]

Hello! You can achieve this in two ways:

Option 1: Orchestrator Job with Task Values
Reference: https://learn.microsoft.com/en-us/azure/databricks/jobs/task-values

Create three separate jobs and one orchestrator job that calls the others. In each transformation, set a variable called job_id and use:

python
dbutils.jobs.taskValues.set(key="first_job_id", value=job_id)

This creates a variable `first_job_id` that can be accessed in other tasks using:

{{tasks.task_for_job_1.values.first_job_id}}

Option 2: Built-in Job Parameters
Reference: https://docs.databricks.com/aws/en/jobs/dynamic-value-references

You can simply use Databricks' built-in dynamic references in your job parameters:

json
{
"parameters": [
{
"name": "my_job_id",
"default": "{{job.id}}"
},
{
"name": "run_date",
"default": "{{job.start_time.iso_date}}"
}
]
}

[dpc]

Thanks. I'll give that a try

[stbjelcevic]

+1 to @jooguilhermesc Option 1 response above.