topic Re: Retrieve job-level parameters in spark_python_task (not notebooks) in Data Engineering

Retrieve job-level parameters in spark_python_task (not notebooks)

xiangzhu — Fri, 21 Jun 2024 10:43:14 GMT

Hello,

I would like to use job parameters in spark_python_task (not notebook_task), does anyone know how to retrieve these parameters inside pure Python ?

I tried:

1/ dbutils.widgets.get("debug"), got error:

com.databricks.dbutils_v1.InputWidgetNotDefined: No input widget named debug is defined

2/ use argparse to define a debug param, but after test, debug is not passed as param:

parser: (Namespace(debug=None), [])

My job has debug param:

Re: Retrieve job-level parameters in spark_python_task (not notebooks)

daniel_sahal — Fri, 21 Jun 2024 11:54:29 GMT

@xiangzhu It looks like it's the same as in https://community.databricks.com/t5/data-engineering/use-job-parameters-in-scripts/td-p/75296

Re: Retrieve job-level parameters in spark_python_task (not notebooks)

xiangzhu — Sun, 23 Jun 2024 18:12:28 GMT

@daniel_sahal

Thanks for your reply. Yes, I have the same question as the thread you mentioned, but argparse doesn't work for job level parameters; it only works for task level parameters.

The Databricks doc is minimal at this level and only says that job level parameters will be propagated to tasks. Nothing more.

Re: Retrieve job-level parameters in spark_python_task (not notebooks)

xiangzhu — Sun, 23 Jun 2024 18:18:21 GMT

During this weekend, I found a workaround by leveraging the databaricks jobs api.

It works but I must provide credentials, whereas I'm looking forward to a solution without credentials since I'm already in a job run.

Re: Retrieve job-level parameters in spark_python_task (not notebooks)

xiangzhu — Sun, 23 Jun 2024 18:24:04 GMT

I rechecked your answner in another thread and sorry I haven't seen `{{job.parameters.[name]}}`, I will try it in the coming days.

Re: Retrieve job-level parameters in spark_python_task (not notebooks)

xiangzhu — Mon, 24 Jun 2024 14:08:47 GMT

@daniel_sahal just tested {{job.parameters.[name]}}, it works, thanks again !