@AmpolJon I don't think you should be giving up on the method, the API allows you to pass job parameters to it and you can retrieve them from in the Python Notebook. Here's an example.

1. Call on the API, https://docs.databricks.com/api/workspace/jobs/runnow 

2. Now, I take a look at the job, and check if the Parameters were passed through. Please note, I didn't set these up as part of the job, this was just through the API call. Also, I forgot to tick "Performance Optimized" in the job config so my job is taking a while longer to run 🙂.

3. Here's the run itself, and it errors, just like yours @AmpolJon 

4. Here's the solution:
Successful job run:

Notebook:

Code (the code comments explain the solution):

# Set a default value to make the widget exist. These will get overwritten anyway! :)
dbutils.widgets.text("param1", "default_value")
dbutils.widgets.text("param2", "default_value")

# Now you can access them (and they'll be overwritten)
p1 = dbutils.widgets.get("param1")
p2 = dbutils.widgets.get("param2")

print(f"param1={p1}, param2={p2}")

Hopefully that resolves your problem @AmpolJon . I'd love to hear more about how you're using N8N and Databricks together as part of your project. If you find the time, it'd be great to share in the Databricks community. I'd love to get some more project-based learning out there.

All the best,
BS