Retrieve job-level parameters in spark_python_task (not notebooks)

xiangzhu
Contributor III

Hello,

I would like to use job parameters in spark_python_task (not notebook_task), does anyone know how to retrieve these parameters inside pure Python ?

xiangzhu_0-1718966193180.png

I tried:

1/ dbutils.widgets.get("debug"), got error: 

    com.databricks.dbutils_v1.InputWidgetNotDefined: No input widget named debug is defined

2/ use argparse to define a debug param, but after test, debug is not passed as param:

    parser: (Namespace(debug=None), [])

My job has debug param:

xiangzhu_1-1718966305318.png