topic Re: Errors calling databricks rest api /api/2.1/jobs/run-now with job_parameters in Data Engineering

Errors calling databricks rest api /api/2.1/jobs/run-now with job_parameters

ErikJ — Fri, 11 Oct 2024 10:10:45 GMT

Hello! I have been using the databricks rest api for running workflows using this endpoint: /api/2.1/jobs/run-now. But now i wanted to also include job_parameters in my api call, i have put job parameters inside my workflow: param1, param2, and in my call to the api im sending this:

params ={"job_id" : 12312312

"job_parameters":{"param1":"12",

"param2":"test"},}. But for some reason i get following response: {'error_code': 'MALFORMED_REQUEST', 'message': "Could not parse request object: Expected 'START_OBJECT' not 'VALUE_STRING'\n at....

Anyone knows what the problem may be? Ive also just copied the sample request as specified in documentation:https://docs.databricks.com/api/azure/workspace/jobs/runnow#job_parameters but its not working either.

Re: Errors calling databricks rest api /api/2.1/jobs/run-now with job_parameters

filipniziol — Fri, 11 Oct 2024 15:03:00 GMT

Hi @ErikJ ,

Are you sending a POST request to the /api/2.1/jobs/run-now endpoint? It's important to use a POST request because this endpoint expects the parameters to be included in the request body as JSON, not as query parameters.

Also, your "params" seem malformed, with a missing comma after "job_id": 12312312 and an extra comma after "job_parameters". This might just be due to copy-pasting, but incorrect JSON formatting can cause the MALFORMED_REQUEST error you're seeing.

Here's how your JSON body should look:

{ "job_id" : 12312312, "job_parameters": { "param1":"12", "param2":"test" } }

Additional Notes:

Ensure Correct HTTP Headers: Make sure you're setting the Content-Type header to application/json in your POST request.
Validate Your JSON: It's a good idea to validate your JSON payload using an online JSON validator to catch any syntax errors.

Re: Errors calling databricks rest api /api/2.1/jobs/run-now with job_parameters

ErikJ — Mon, 14 Oct 2024 10:42:14 GMT

{ "job_id": , "job_parameters": { "params1": "", "params2": "" } }

Yes my bad maybe, but it is correctly formatted, and im using POST exactly as described in the databricks rest api documentation. I even copied their example json but that didn't help. I just know that, i can run the job using api, but as soon as i add parameters inside "job_parameters" i get this in return. But I will look into it today!

Re: Errors calling databricks rest api /api/2.1/jobs/run-now with job_parameters

RyanBarrus — Fri, 18 Oct 2024 19:37:51 GMT

I just had the same error. The cause of the problem for me was that I didn't have depends_on formatted corretly. This may not be the same problem for you, but it might be worth checking similar things. The good news is that it won't spoil your adventure with any unwanted hints in the error message.

I was using: "depends_on": ["A"]
It was fixed by switching to: "depends_on": [{"task_key":"A"}]

Re: Errors calling databricks rest api /api/2.1/jobs/run-now with job_parameters

Panda — Mon, 21 Oct 2024 12:40:18 GMT

@ErikJ I ran into a similar issue before—it was like calling the wrong twin by their sibling’s name! Instead of using job_parameters, I accidentally charmed my way through with notebook_params, and voila, it worked:

params = { "job_id": 12312312, "notebook_params": { "param1": "12", "param2": "test" } }

So, lesson learned: If something isn’t working, maybe it’s not a bug—it’s just your parameters playing hide-and-seek. Always good to try the other twin!

Re: Errors calling databricks rest api /api/2.1/jobs/run-now with job_parameters

-werners- — Mon, 21 Oct 2024 12:59:40 GMT

notebook_params is deprecated in api 2.1:
https://docs.databricks.com/api/workspace/jobs/runnow

Re: Errors calling databricks rest api /api/2.1/jobs/run-now with job_parameters

ErikJ — Mon, 21 Oct 2024 13:06:29 GMT

Hello and thank you for the replies! I found the issue. For some reason I had params = params in my requests.post(), but I swapped to json = params like this:

requests.post(url, headers=headers,json=params).

Re: Errors calling databricks rest api /api/2.1/jobs/run-now with job_parameters

slkdfuba — Fri, 17 Jan 2025 00:28:08 GMT

I encountered a null job_id in my post, when a notebook parameter was set in the job GUI. But it runs just fine (I get a valid job_id with active run) if I delete the notebook parameter in the job GUI.

Is this a documented behavior, or a bug? If it's expected, where is this documented?