cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Retrieve job-level parameters in Python

John_Rotenstein
New Contributor II

Parameters can be passed to Tasks and the values can be retrieved with:

dbutils.widgets.get("parameter_name")

More recently, we have been given the ability to add parameters to Jobs.

However, the parameters cannot be retrieved like Task parameters.

Question: How can we retrieve job-level parameters in notebook code?

 

9 REPLIES 9

Kaniz_Fatma
Community Manager
Community Manager

Hi @John_RotensteinJob-level parameters in notebook code can be retrieved using the dbutils.jobs.taskValues.get() command.

This command allows tasks to output values that can be referenced in subsequent tasks within a job run. 

For instance, if you have two notebook tasks, Get_user_data and Analyze_user_data, and you want to pass a user's name and age from the Get_user_data task to the Analyze_user_data task, you can use the dbutils.jobs.taskValues.set() to set these values in the Get_user_data task and then use dbutils.jobs.taskValues.get() to retrieve these values in the Analyze_user_data task.

Here is an example of how to develop and retrieve task values:

python
# Set task values in the 'Get_user_data' task
dbutils.jobs.taskValues.set(key = 'name', value = 'Some User')
dbutils.jobs.taskValues.set(key = "age", value = 30)

# Get task values in the 'Analyze_user_data' task
dbutils.jobs.taskValues.get(taskKey = "Get_user_data", key = "age", default = 42, debugValue = 0)
dbutils.jobs.taskValues.get(taskKey = "Get_user_data", key = "name", default = "Jane Doe")

In the dbutils.jobs.taskValues.get() command, the taskKey is the name of the job task setting the value, the key is the name of the task value's key, default is an optional value that is returned if key cannot be found, and debugValue It is an optional value that is returned if you try to get the task value from within a notebook running outside a job.

cbern
New Contributor III

@Kaniz_Fatma This method works for Task parameters. Is there a way to access Job parameters that apply to the entire workflow, set under a heading like this in the UI:

cbern_0-1714069648952.png

I am able to read Job parameters in a different way from Task parameters using  dynamic value references:

{{tasks.[task_name].values.[value_name]}}

vs.

{{job.parameters.[name]}} 

This works for reading Parameters in the Workflow itself, such as:

cbern_1-1714070664308.png

Is there an analogous way to read a Job parameter within a Notebook? The Note on this page seems to indicate that these dynamic value references are available in Notebooks, but how do you reference them in Python?

cbern
New Contributor III

A coworker has answered this question for me, posting it for anyone else looking for an answer:

run_parameters = dbutils.notebook.entry_point.getCurrentBindings() 
context = json.loads(dbutils.notebook.entry_point.getDbutils().notebook().getContext().toJson())
job_param = run_parameters[key]

cbern
New Contributor III

 

an update to my answer: Databricks has advised us that the `dbutils.notebook.entry_point` method is not supported (could be deprecated), and the recommended way to read in a job parameter is through widgets, i.e. `dbutils.widgets.get("param_key")` (similar to Task parameters -- if you have a Task param and Job param with the same name, the Job param value takes precedence). 

If you want to read a dynamic value reference like the run_id inside a Notebook, you can set the reference (e.g. `{{job.run_id}}`) a Task or Job param, and read it like a widget.

xiangzhu
Contributor III

I think `dbutils.widgets.get` is for notebooks only, not for python job as asked by this thread

xiangzhu
Contributor III

ah sorry, the thread asked for notebooks too.

nevertheless, I'm search for getting job params in pure python jobs

You need to push down your parameters to a task level. Eg:

  1. Create a job level parameter called "my_param"
  2. Make a reference to his job parameter in the task level parameters box. Eg:
    ["--my_param","{{job.parameters.my_param}}"]
  3. Read the task level parameter using argparser in your .py file

Thanks !

lprevost
Contributor

The only thing that has worked for me consistently in python is 

params = dbutils.widgets.getAll() where an empty dictionary is returned if I'm in interactive mode and the job/task params are returned if they are present.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group