Pass parameters between jobs

[dpc]

Hello

I have job

In that job, it runs a task (GetGid) that executes a notebook and obtains some value using dbutils.jobs.taskValuesSet

e.g. dbutils.jobs.taskValuesSet(key = "gid", value = gid)

As a result, I can use this and pass it to another task for use by setting the parameter

e.g. Key - gid; Value - {{tasks.GetGid.values.gid}}

This works find and it see gid and uses it (the task runs a sql notebook with a widget

How do I pass and use this to a job task?

gid should be passed such that it can be used in tasks there

e.g. a futher notebook task or maybe an if/then task

All within the downstream job

I can do this easily in Azure Data Factory but cannot find a solution in databricks

Any ideas?

Is this even possible? It feels fairly basic

Thanks

[ilir_nuredini]

Hi @dpc ,

I am having a bit difficulty to understand complete what is your need. May you check this link: https://docs.databricks.com/aws/en/jobs/task-values  and see if that resolves your need. Let me know how it goes.

Best, Ilir

[dpc]

Thanks.

That works fine but what I really want to be able to do is pass parameters between jobs.

So, create a job task within a job and pass all the parameters down to that job so they can be used

[Hubert-Dudek · Accepted solution]

Python:
dbutils.jobs.taskValuesSet()

SQL:

Results from sql query are visible in the workflow and you can use them to orchestrate another task (first row or loop)

[antbuono]

What about SQL variables? Can I retrieve in a latter task a variable set in a previous one?

[dpc]

Thanks @Hubert-Dudek and @ilir_nuredini 

I see this now

I'm setting using:dbutils.jobs.taskValues.Set()

passing to the job task using Key - gid; Value - {{tasks.GetGid.values.gid}}

Then reading using: pid = dbutils.widgets.get()