topic Re: Allowing a job parameter that is pushed down to be overridden in Data Engineering

Allowing a job parameter that is pushed down to be overridden

dpc — Fri, 13 Mar 2026 10:42:40 GMT

Hello

I have a Job that calls a job which calls a job (this could go on)

I want to generate an id for each job and log that id along with a parent id and job name

So, I am creating an id at each level as the first task

Then passing this to the next level as the ParentId

At the next level it then generates and id and would then log the parent (pushed) and id

So (e.g):

The top 2 levels are fine but when I get to the 3rd (and beyond),

I cannot pass the Level 2 Id as the parent because it's already defined and will not let me override

Is this possible?

I can do this in Azure Data Factory but want to replace the functionality

Thanks

jooguilhermesc — Fri, 13 Mar 2026 12:00:45 GMT

Hello! You can achieve this in two ways:

Create three separate jobs and one orchestrator job that calls the others. In each transformation, set a variable called job_id and use:

python dbutils.jobs.taskValues.set(key="first_job_id", value=job_id)

This creates a variable `first_job_id` that can be accessed in other tasks using:

You can simply use Databricks' built-in dynamic references in your job parameters:

json { "parameters": [ { "name": "my_job_id", "default": "{{job.id}}" }, { "name": "run_date", "default": "{{job.start_time.iso_date}}" } ] }

dpc — Sun, 15 Mar 2026 11:20:12 GMT

Thanks. I'll give that a try

stbjelcevic — Mon, 23 Mar 2026 00:38:28 GMT

+1 to @jooguilhermesc Option 1 response above.