topic Re: Retrieve job-level parameters in Python in Data Engineering

Retrieve job-level parameters in Python

John_Rotenstein — Thu, 14 Sep 2023 08:44:25 GMT

Parameters can be passed to Tasks and the values can be retrieved with:

dbutils.widgets.get("parameter_name")

More recently, we have been given the ability to add parameters to Jobs.

However, the parameters cannot be retrieved like Task parameters.

Question: How can we retrieve job-level parameters in notebook code?

Re: Retrieve job-level parameters in Python

cbern — Thu, 25 Apr 2024 18:46:00 GMT

@Retired_mod This method works for Task parameters. Is there a way to access Job parameters that apply to the entire workflow, set under a heading like this in the UI:

I am able to read Job parameters in a different way from Task parameters using dynamic value references:

{{tasks.[task_name].values.[value_name]}}

vs.

{{job.parameters.[name]}}

This works for reading Parameters in the Workflow itself, such as:

Is there an analogous way to read a Job parameter within a Notebook? The Note on this page seems to indicate that these dynamic value references are available in Notebooks, but how do you reference them in Python?

Re: Retrieve job-level parameters in Python

cbern — Thu, 25 Apr 2024 19:10:58 GMT

A coworker has answered this question for me, posting it for anyone else looking for an answer:

run_parameters = dbutils.notebook.entry_point.getCurrentBindings() 
context = json.loads(dbutils.notebook.entry_point.getDbutils().notebook().getContext().toJson()) 
job_param = run_parameters[key]

Re: Retrieve job-level parameters in Python

cbern — Fri, 03 May 2024 14:49:45 GMT

an update to my answer: Databricks has advised us that the `dbutils.notebook.entry_point` method is not supported (could be deprecated), and the recommended way to read in a job parameter is through widgets, i.e. `dbutils.widgets.get("param_key")` (similar to Task parameters -- if you have a Task param and Job param with the same name, the Job param value takes precedence).

If you want to read a dynamic value reference like the run_id inside a Notebook, you can set the reference (e.g. `{{job.run_id}}`) a Task or Job param, and read it like a widget.

Re: Retrieve job-level parameters in Python

xiangzhu — Fri, 21 Jun 2024 10:08:33 GMT

I think `dbutils.widgets.get` is for notebooks only, not for python job as asked by this thread

Re: Retrieve job-level parameters in Python

xiangzhu — Fri, 21 Jun 2024 10:31:20 GMT

ah sorry, the thread asked for notebooks too.

nevertheless, I'm search for getting job params in pure python jobs

Re: Retrieve job-level parameters in Python

julio_resende — Fri, 12 Jul 2024 20:51:24 GMT

You need to push down your parameters to a task level. Eg:

Create a job level parameter called "my_param"
Make a reference to his job parameter in the task level parameters box. Eg:
["--my_param","{{job.parameters.my_param}}"]
Read the task level parameter using argparser in your .py file

Re: Retrieve job-level parameters in Python

xiangzhu — Fri, 12 Jul 2024 21:36:06 GMT

Thanks !

Re: Retrieve job-level parameters in Python

lprevost — Tue, 06 Aug 2024 21:01:36 GMT

The only thing that has worked for me consistently in python is

params = dbutils.widgets.getAll() where an empty dictionary is returned if I'm in interactive mode and the job/task params are returned if they are present.

Re: Retrieve job-level parameters in Python

DineshOjha — Fri, 31 Jan 2025 22:49:11 GMT

How do I read the task level parameter using argparser?

Re: Retrieve job-level parameters in Python

xiangzhu — Sat, 01 Feb 2025 14:16:29 GMT

@DineshOjha

The task-level parameter is not very useful, in my opinion, because it is hardcoded and not a real parameter. In such cases, I often use a config.py file to define all task level parameters directly within Python as a configuration.

However, job-level parameters are really useful since we can change their values dynamically when manually triggering a job run.

I'm not aware if Databricks' dbutils provides a built-in method to directly return the current job parameters, but we can work around this by querying the Jobs API.

At the very beginning of the task's Python code:

Use dbutils to get the current job run ID.
Use the Jobs API to retrieve the current job info using the job run ID. (whether from raw api call, or from the databricks_cli sdk, not the new go lib, but the old python one)
Extract the job_parameters from the returned job info.
Save all the job_parameters as environment variables.

From anywhere in your code, you can now access the current job parameters from the environment variables without using argparse.

For steps 1–4, you could write a function to encapsulate the entire process.