โ07-31-2024 04:07 AM
I have workflow like this.
1 task: job as a task. Inside this job there is task which is seting parameter x as taskValue using dbutils.jobs.taskValues.set.
2. task dependent on previous job as a task. I would like to access this parameter x. I tried to do dynamic job parameter like {{tasks.task1.x}} and then dbutils.widgets.get(x) but this taskValue is not detected in dynamic parameters.
Any Idea how could I pass this taskValue or make it job parameter?
โ07-31-2024 05:11 PM
Hi,
Can you please try dbutils.jobs.taskValues.get()
https://docs.databricks.com/en/workflows/jobs/share-task-context.html#using-task-values
Thanks!
โ07-31-2024 05:13 PM
https://docs.databricks.com/en/dev-tools/databricks-utils.html#dbutils-jobs-taskvalues-set
dbutils.jobs.taskValues.set(key = 'name', value = 'Some User')
dbutils.jobs.taskValues.set(key = "age", value = 30)
https://docs.databricks.com/en/dev-tools/databricks-utils.html#get-command-dbutilsjobstaskvaluesget
dbutils.jobs.taskValues.get(taskKey = "my-task", \
key = "my-key", \
default = 7, \
debugValue = 42)
dbutils.jobs.taskValues.get(taskKey = "my-task", \
key = "my-key", \
default = 7, \
debugValue = 42)
dbutils.jobs.taskValues.get(taskKey = "my-task", \
key = "my-key", \
default = 7, \
debugValue = 42)
โ08-01-2024 06:04 AM
I tried this approach and it only works when I am passing value from normal notebook task to another task / job. But when I have "job as task" as source, then it is not working. Maybe I am missing something.
I did experiment with 2 tasks passing parameters to destination task. First task was normal notebook task, second was job as a task and it did not work there.
โ08-02-2024 06:16 PM
I see, I have requested for someone else to guide you on this.
cc: @Retired_mod
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.
Request a New Group