Retrieve job-level parameters in spark_python_task (not notebooks)

[xiangzhu]

Hello,

I would like to use job parameters in spark_python_task (not notebook_task), does anyone know how to retrieve these parameters inside pure Python ?

I tried:

1/ dbutils.widgets.get("debug"), got error: 

    com.databricks.dbutils_v1.InputWidgetNotDefined: No input widget named debug is defined

2/ use argparse to define a debug param, but after test, debug is not passed as param:

    parser: (Namespace(debug=None), [])

My job has debug param:

[daniel_sahal · Accepted solution]

@xiangzhu It looks like it's the same as in https://community.databricks.com/t5/data-engineering/use-job-parameters-in-scripts/td-p/75296

[xiangzhu]

@daniel_sahal 

Thanks for your reply. Yes, I have the same question as the thread you mentioned, but argparse doesn't work for job level parameters; it only works for task level parameters.

The Databricks doc is minimal at this level and only says that job level parameters will be propagated to tasks. Nothing more.

[xiangzhu]

During this weekend, I found a workaround by leveraging the databaricks jobs api.

It works but I must provide credentials, whereas I'm looking forward to a solution without credentials since I'm already in a job run.

[xiangzhu]

I rechecked your answner in another thread and sorry I haven't seen `{{job.parameters.[name]}}`, I will try it in the coming days.

[xiangzhu]

@daniel_sahal just tested {{job.parameters.[name]}}, it works, thanks again !