topic Re: Get taskValue from job as task, and then pass it to next task. in Data Engineering

Get taskValue from job as task, and then pass it to next task.

CaptainJack — Wed, 31 Jul 2024 11:07:34 GMT

I have workflow like this.

1 task: job as a task. Inside this job there is task which is seting parameter x as taskValue using dbutils.jobs.taskValues.set.

2. task dependent on previous job as a task. I would like to access this parameter x. I tried to do dynamic job parameter like {{tasks.task1.x}} and then dbutils.widgets.get(x) but this taskValue is not detected in dynamic parameters.

Any Idea how could I pass this taskValue or make it job parameter?

Re: Get taskValue from job as task, and then pass it to next task.

NandiniN — Thu, 01 Aug 2024 00:11:09 GMT

Hi,

Can you please try dbutils.jobs.taskValues.get()

https://docs.databricks.com/en/workflows/jobs/share-task-context.html#using-task-values

Thanks!

Re: Get taskValue from job as task, and then pass it to next task.

NandiniN — Thu, 01 Aug 2024 00:13:28 GMT

https://docs.databricks.com/en/dev-tools/databricks-utils.html#dbutils-jobs-taskvalues-set

dbutils.jobs.taskValues.set(key = 'name', value = 'Some User')
dbutils.jobs.taskValues.set(key = "age", value = 30)

https://docs.databricks.com/en/dev-tools/databricks-utils.html#get-command-dbutilsjobstaskvaluesget

dbutils.jobs.taskValues.get(taskKey = "my-task", \
key = "my-key", \
default = 7, \
debugValue = 42)

Re: Get taskValue from job as task, and then pass it to next task.

CaptainJack — Thu, 01 Aug 2024 13:04:55 GMT

I tried this approach and it only works when I am passing value from normal notebook task to another task / job. But when I have "job as task" as source, then it is not working. Maybe I am missing something.

I did experiment with 2 tasks passing parameters to destination task. First task was normal notebook task, second was job as a task and it did not work there.

Re: Get taskValue from job as task, and then pass it to next task.

NandiniN — Sat, 03 Aug 2024 01:16:34 GMT

I see, I have requested for someone else to guide you on this.

cc: @Retired_mod