topic Re: Read task level parameters in python in Data Engineering

Read task level parameters in python

DineshOjha — Fri, 31 Jan 2025 22:56:03 GMT

I am creating Databricks jobs and tasks using a python package. I have defined a task level parameter and would like to reference it in my script using sys.argv. How can I do that ?

Re: Read task level parameters in python

brockb — Sat, 01 Feb 2025 04:08:10 GMT

Hi @DineshOjha ,

You can reference the task-level parameters using `my_python_var = dbutils.widgets.get('my-task-level-param')`.

More information on task values can be found here: https://docs.databricks.com/en/jobs/task-values.html#set-task-values.

Hope this helps.

Re: Read task level parameters in python

DineshOjha — Mon, 03 Feb 2025 20:41:37 GMT

Thanks, but the link works for notebook. I have a python package run as python wheel and am wondering how to access the parameters. When I run the job its not able to understand the task level parameters in sys.argv