topic Re: Rest API invocation for databricks notebook fails while invoking from ADF pipeline in Data Engineering

Rest API invocation for databricks notebook fails while invoking from ADF pipeline

sparkstreaming — Mon, 16 May 2022 12:58:01 GMT

In the current implementation a streaming databricks notebook needs to be started based on the configuration passed. Since the rest of databricks notebooks are being invoked by using ADF,it was decided to use ADF for starting these notebooks. Since there are followup activites that needs to be done after the notebook starts, we tried to start the streaming notebook from an ADF pipeline VIA Rest API.The ADF WEB component is being leveraged to call the rest api for running a notebook. However while trying to invoke the notebook we are getting a Malformed URL Exception :-

{"error_code":"MALFORMED_REQUEST","message":"Invalid JSON given in the body of the request - failed to parse given JSON"}

The below JSON Snippet is being passed as the body for the request

{
  "tasks": [
    {
      "task_key": "Match",
      "description": "Matches orders with user sessions",
      "notebook_task": {
        "notebook_path": "/Users/userxxx@xxxxsandbox.com/Demo/RealTimeXXXXXXX",
      },
      "timeout_seconds": 86400
    }
  ],
  "run_name": "A multitask job run",
  "git_source": null,
  "timeout_seconds": 86400
}

Can somebody guide me on what am I doing wrong here

Re: Rest API invocation for databricks notebook fails while invoking from ADF pipeline

Hubert-Dudek — Tue, 17 May 2022 09:48:06 GMT

@Prasanth KP ,

remove comma after XXXXXXX",
remove "git_source": null
in ADF, I had to wrap everything in single quotes
Please test your API call with curl or postman first.

'{
  "tasks": [
    {
      "task_key": "Match",
      "description": "Matches orders with user sessions",
      "notebook_task": {
        "notebook_path": "/Users/userxxx@xxxxsandbox.com/Demo/RealTimeXXXXXXX"
      },
      "timeout_seconds": 86400
    }
  ],
  "run_name": "A multitask job run",
  "timeout_seconds": 86400
}'

Re: Rest API invocation for databricks notebook fails while invoking from ADF pipeline

-werners- — Tue, 17 May 2022 09:54:52 GMT

@Prasanth KP ,

clearly, the rest call is invalid. What endpoint do you call?

Also do not forget to authenticate.

May I ask why you use the REST API instead of the available notebook functionality of ADF?

Re: Rest API invocation for databricks notebook fails while invoking from ADF pipeline

sparkstreaming — Thu, 19 May 2022 05:05:33 GMT

Thanks Hubert. This worked.

Re: Rest API invocation for databricks notebook fails while invoking from ADF pipeline

sparkstreaming — Thu, 19 May 2022 05:06:29 GMT

Thank you Kaniz. The solution suggested by Hubert worked.