How to sending parameters from http request to in job running notebook

[AmpolJon]

I've try to trigger job running via n8n workflow, which can command to make notebook running properly.

BUT another bullet to achieve is I have to send some data to that job to be rub as well, i googled it and can't find solutions anywhere. My setup was

Method : POST
URL : https://abcd.databricks.com/api/2.2/jobs/run-now
json attached to request :
{
"job_id" : "569943055844936",
"job_parameters": {
"param1": "string1",
"param2": "string2"
},
"notebook_params": {
"input_path": "dbfs:/mnt/data/train.csv",
"num_epochs": "20"
}
}

In Python Notebook

    params = dbutils.widgets.getAll()

    print(params)
it return nothing

[szymon_dybczak · Accepted solution]

Hi @AmpolJon ,

Passing parameters  is not supported by this API. That's why when you're defining notebook_params they'are ignored:
Trigger a new job run | Jobs API | REST API reference | Databricks on AWS

You can check following workaround though. The user is using update API to set task parameters and then executes the job. Not the most beautiful solution in the world but it should work:

https://stackoverflow.com/a/75607277

[szymon_dybczak]

To be precise - passing parameters at task level is not supported by this API.

[szymon_dybczak]

Or you can consider using job parameters instead of task parameter in this case 🙂

[BS_THE_ANALYST · Accepted solution]

@AmpolJon the job parameter should do the trick as @szymon_dybczak mentions. It's supported in the docs

Just be mindful that job parameters take precedence over task parameters. In your case though, task parameters won't work through the API, so configure your solution to leverage the job parameters 🙂.

Please feedback how you get on.

All the best,
BS

[AmpolJon]

I've try passing the job_parameters as
"job_parameters": {
  "param1": "string1",
  "param2": "string2"
}

and wishing to reach string1 in python using this code

param = dbutils.widgets.get("param1")

it return error as picture attached, i just want to assure that I should gave up on this method or I'm about in the right path?

[BS_THE_ANALYST]

@AmpolJon I don't think you should be giving up on the method, the API allows you to pass job parameters to it and you can retrieve them from in the Python Notebook. Here's an example.

1. Call on the API, https://docs.databricks.com/api/workspace/jobs/runnow 

2. Now, I take a look at the job, and check if the Parameters were passed through. Please note, I didn't set these up as part of the job, this was just through the API call. Also, I forgot to tick "Performance Optimized" in the job config so my job is taking a while longer to run 🙂.

3. Here's the run itself, and it errors, just like yours @AmpolJon 

4. Here's the solution:
Successful job run:

Notebook:

Code (the code comments explain the solution):

# Set a default value to make the widget exist. These will get overwritten anyway! :)
dbutils.widgets.text("param1", "default_value")
dbutils.widgets.text("param2", "default_value")

# Now you can access them (and they'll be overwritten)
p1 = dbutils.widgets.get("param1")
p2 = dbutils.widgets.get("param2")

print(f"param1={p1}, param2={p2}")

Hopefully that resolves your problem @AmpolJon . I'd love to hear more about how you're using N8N and Databricks together as part of your project. If you find the time, it'd be great to share in the Databricks community. I'd love to get some more project-based learning out there.

All the best,
BS