topic 'dbutils.jobs.taskValues.get' taking debug value in workflow, instead of actual value being set in Data Engineering

'dbutils.jobs.taskValues.get' taking debug value in workflow, instead of actual value being set

KalyaniJaya — Thu, 23 Jan 2025 11:19:37 GMT

Hi,

I am trying to pass and set values from one wheel into another wheel in databricks workflow.

I have used 'dbutils.jobs.taskValues.get' and 'dbutils.jobs.taskValues.set'

I have used 'dbutils.jobs.taskValues.get' in second task and made sure to keep dependency of task 1 wheel to complete before task 2 kicks in.

The issue is as below :

while trying to get the task key , in workflow by default it is considering the debug value , instead of actual value.

value = 'dbutils.jobs.taskValues.get'(taskKey="BatchProcessing", key ="batch_result",default=None,

debugValue="samplee")

could anyone help in fixing the issue.

Re: 'dbutils.jobs.taskValues.get' taking debug value in workflow, instead of actual value being set

Alberto_Umana — Thu, 23 Jan 2025 12:42:58 GMT

It seems like the issue you're encountering is related to the debugValue parameter being used instead of the actual value when calling dbutils.jobs.taskValues.get. This behavior is expected when the notebook is run outside of a job context, as the debugValue is returned in such cases

Please try this approach:

Task 1: Setting the value

dbutils.jobs.taskValues.set(key="batch_result", value="actual_value")

Task 2: Getting the value

value = dbutils.jobs.taskValues.get(taskKey="BatchProcessing", key="batch_result", default=None)

print(value) # This should print "actual_value" if running within a job context