Retrieve job-level parameters in Python
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-14-2023 01:44 AM
Parameters can be passed to Tasks and the values can be retrieved with:
More recently, we have been given the ability to add parameters to Jobs.
However, the parameters cannot be retrieved like Task parameters.
Question: How can we retrieve job-level parameters in notebook code?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-25-2024 11:46 AM
@Retired_mod This method works for Task parameters. Is there a way to access Job parameters that apply to the entire workflow, set under a heading like this in the UI:
I am able to read Job parameters in a different way from Task parameters using dynamic value references:
{{tasks.[task_name].values.[value_name]}}
vs.
{{job.parameters.[name]}}
This works for reading Parameters in the Workflow itself, such as:
Is there an analogous way to read a Job parameter within a Notebook? The Note on this page seems to indicate that these dynamic value references are available in Notebooks, but how do you reference them in Python?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
04-25-2024 12:10 PM
A coworker has answered this question for me, posting it for anyone else looking for an answer:
run_parameters = dbutils.notebook.entry_point.getCurrentBindings()
context = json.loads(dbutils.notebook.entry_point.getDbutils().notebook().getContext().toJson())
job_param = run_parameters[key]
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-03-2024 07:49 AM
an update to my answer: Databricks has advised us that the `dbutils.notebook.entry_point` method is not supported (could be deprecated), and the recommended way to read in a job parameter is through widgets, i.e. `dbutils.widgets.get("param_key")` (similar to Task parameters -- if you have a Task param and Job param with the same name, the Job param value takes precedence).
If you want to read a dynamic value reference like the run_id inside a Notebook, you can set the reference (e.g. `{{job.run_id}}`) a Task or Job param, and read it like a widget.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-21-2024 03:08 AM
I think `dbutils.widgets.get` is for notebooks only, not for python job as asked by this thread
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-21-2024 03:31 AM
ah sorry, the thread asked for notebooks too.
nevertheless, I'm search for getting job params in pure python jobs
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-12-2024 01:51 PM
You need to push down your parameters to a task level. Eg:
- Create a job level parameter called "my_param"
- Make a reference to his job parameter in the task level parameters box. Eg:["--my_param","{{job.parameters.my_param}}"]
- Read the task level parameter using argparser in your .py file
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-12-2024 02:36 PM
Thanks !
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-06-2024 02:01 PM
The only thing that has worked for me consistently in python is