cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

'dbutils.jobs.taskValues.get' taking debug value in workflow, instead of actual value being set

KalyaniJaya
New Contributor

Hi,

I am trying to pass and set values from one wheel into another wheel in databricks workflow.

I have used 'dbutils.jobs.taskValues.get' and 'dbutils.jobs.taskValues.set'

I have used 'dbutils.jobs.taskValues.get' in second task and made sure to keep dependency of task 1 wheel to complete before task 2 kicks in. 

The issue is as below :

while trying to get the task key , in workflow by default it is considering the debug value , instead of actual value.

value = 'dbutils.jobs.taskValues.get'(taskKey="BatchProcessing", key ="batch_result",default=None,

        debugValue="samplee")
could anyone help in fixing the issue.
1 REPLY 1

Alberto_Umana
Databricks Employee
Databricks Employee

It seems like the issue you're encountering is related to the debugValue parameter being used instead of the actual value when calling dbutils.jobs.taskValues.get. This behavior is expected when the notebook is run outside of a job context, as the debugValue is returned in such cases

Please try this approach:

Task 1: Setting the value

dbutils.jobs.taskValues.set(key="batch_result", value="actual_value")

 

Task 2: Getting the value

value = dbutils.jobs.taskValues.get(taskKey="BatchProcessing", key="batch_result", default=None)

print(value)  # This should print "actual_value" if running within a job context

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group