topic Pass parameters between jobs in Data Engineering

Pass parameters between jobs

dpc — Sat, 25 Oct 2025 16:38:43 GMT

Hello

I have job

In that job, it runs a task (GetGid) that executes a notebook and obtains some value using dbutils.jobs.taskValuesSet

e.g. dbutils.jobs.taskValuesSet(key = "gid", value = gid)

As a result, I can use this and pass it to another task for use by setting the parameter

e.g. Key - gid; Value - {{tasks.GetGid.values.gid}}

This works find and it see gid and uses it (the task runs a sql notebook with a widget

How do I pass and use this to a job task?

gid should be passed such that it can be used in tasks there

e.g. a futher notebook task or maybe an if/then task

All within the downstream job

I can do this easily in Azure Data Factory but cannot find a solution in databricks

Any ideas?

Is this even possible? It feels fairly basic

Thanks

Re: Pass parameters between jobs

ilir_nuredini — Sun, 26 Oct 2025 14:48:33 GMT

Hi @dpc ,

I am having a bit difficulty to understand complete what is your need. May you check this link: https://docs.databricks.com/aws/en/jobs/task-values and see if that resolves your need. Let me know how it goes.

Best, Ilir

Re: Pass parameters between jobs

dpc — Sun, 26 Oct 2025 18:36:17 GMT

Thanks.

That works fine but what I really want to be able to do is pass parameters between jobs.

So, create a job task within a job and pass all the parameters down to that job so they can be used

Re: Pass parameters between jobs

Hubert-Dudek — Sun, 26 Oct 2025 19:46:48 GMT

Python:
dbutils.jobs.taskValuesSet()

SQL:

Results from sql query are visible in the workflow and you can use them to orchestrate another task (first row or loop)

Re: Pass parameters between jobs

dpc — Mon, 27 Oct 2025 09:56:58 GMT

Thanks @Hubert-Dudek and @ilir_nuredini

I see this now

I'm setting using:dbutils.jobs.taskValues.Set()

passing to the job task using Key - gid; Value - {{tasks.GetGid.values.gid}}

Then reading using: pid = dbutils.widgets.get()

Re: Pass parameters between jobs

antbuono — Tue, 28 Oct 2025 14:25:08 GMT

What about SQL variables? Can I retrieve in a latter task a variable set in a previous one?