cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Share information between tasks in a Databricks job  You can use task values to pass arbitrary parameters between tasks in a Databricks job. You pass ...

Ajay-Pandey
Esteemed Contributor III

Share information between tasks in a Databricks job

You can use task values to pass arbitrary parameters between tasks in a Databricks job. You pass task values using the taskValues subutility in Databricks Utilities. The taskValues subutility provides a simple API that allows tasks to output values that can be referenced in subsequent tasks, making it easier to create more expressive workflows. For example, you can communicate identifiers or metrics, such as information about the evaluation of a machine learning model, between different tasks within a job run. Each task can set and get multiple task values. Task values can be set and retrieved in Python notebooks.

The taskValues subutility provides two commands: dbutils.jobs.taskValues.set() to set a variable and dbutils.jobs.taskValues.get() to retrieve a value. Suppose you have two notebook tasks: Get_user_data and Analyze_user_data and want to pass a user’s name and age from the Get_user_data task to the Analyze_user_data task. The following example sets the user’s name and age in the Set_user_data task:

##Set task value
dbutils.jobs.taskValues.set(key = 'name', value = 'Some User')
 
##Get task value
dbutils.jobs.taskValues.get(taskKey = "Get_user_data", key = "name", default = "Jane Doe")

Ajay Kumar Pandey
2 REPLIES 2

jose_gonzalez
Databricks Employee
Databricks Employee

Thank you for sharing this

newforesee
New Contributor II

We urgently hope for this feature, but to date, we have found that it is only available in Python. Do you have any plans to support Scala?

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group