topic Re: Support for kwargs parameter in `/2.1/jobs/create` endpoint for `python_wheel_task` in Data Engineering

Support for kwargs parameter in `/2.1/jobs/create` endpoint for `python_wheel_task`

GabrieleMuciacc — Mon, 14 Feb 2022 17:30:11 GMT

If I create a job from the web UI and I select Python wheel, I can add kwargs parameters. Judging from the generated JSON job description, they appear under a section named `namedParameters`.

However, if I use the REST APIs to create a job, it appears I cannot pass namedParameters (link to the documentation).

Am I right that it is not possible to create a Python wheel task with kwargs parameter using REST APIs? Is there any plan to add support?

For context, our use case is that we create jobs using our CI, but we sometimes want to run them manually with different parameters, and we'd like to have the nice kwargs UI to modify the parameters.

Re: Support for kwargs parameter in `/2.1/jobs/create` endpoint for `python_wheel_task`

GabrieleMuciacc — Wed, 23 Feb 2022 13:38:09 GMT

Hi @Kaniz Fatma, thanks for your answer. I do not understand what is the correlation between the invoke library and Databricks REST APIs. Could you please elaborate?

Re: Support for kwargs parameter in `/2.1/jobs/create` endpoint for `python_wheel_task`

rajeev_thakur_c — Wed, 16 Mar 2022 08:44:23 GMT

Hi there, the documentation is not up-to-date, you should be able to create such a task with

namedParameters instead of parameters

. Here is a small example:

{

"name": "test_entry_point",

"tasks": [

{

"task_key": "test_entry_point",

"description": "test_entry_point",

"existing_cluster_id": "an-interactive-cluster-that-I-should-probably-not-use",

"python_wheel_task": {

"package_name": "my_package_name",

"entry_point": "my_entry_point",

"named_parameters": {

"param-1": "TODO",

"param-2": "TODO"

}

"libraries": [

{

"whl": "dbfs:/TODO"

}

]

}

]

}

I am flagging the fact that the doc is not up-to-date. We will update it soon.

Re: Support for kwargs parameter in `/2.1/jobs/create` endpoint for `python_wheel_task`

GabrieleMuciacc — Wed, 16 Mar 2022 11:45:09 GMT

Hi @Rajeev Kumar , I've tried and it works, thanks!

Re: Support for kwargs parameter in `/2.1/jobs/create` endpoint for `python_wheel_task`

rajeev_thakur_c — Wed, 16 Mar 2022 11:46:57 GMT

happy to help.

Great Day Ahead!

Re: Support for kwargs parameter in `/2.1/jobs/create` endpoint for `python_wheel_task`

manojpatil04 — Wed, 11 Dec 2024 05:08:56 GMT

@GabrieleMuciacc , in case of serverless compute job this can be pass as external dependency you can't use libraries.

"tasks": [{ "task_key": task_id,

"spark_python_task": {

"python_file": py_file,

"parameters": parameters

"environment_key": "Test_environment" }],

"environments": [

{

"environment_key": "Test_environment",

"spec":

{

"client": "1",

"dependencies":["wheel file path preferable on volumes"]

}

}],