Databricks Jobs API not returning notebook run results?

apayne — Wed, 23 Nov 2022 17:32:31 GMT

Calling a databricks notebook using the Rest API, can confirm that it is executing the notebook, but is not accepting my parameters or returning a notebook output. Any ideas on what I am doing wrong here?

My code and notebook function are below, trying to pass a string to the notebook function and have it return the results.

# Databricks Rest API calls from IDE
 
import requests
import json
import time
 
job_payload = {
  "run_name": 'execute_test',
  "existing_cluster_id": '<cluster-id>',
  "notebook_task":
    {
      "notebook_path": '/Users/<user>/testing',
      "source": "WORKSPACE",
      "base_parameters": {"data": "test"}
    }
}
 
resp = requests.post('https://<url>/api/2.1/jobs/runs/submit', json=job_payload, headers={'Authorization': 'Bearer <access token>'})
 
run_id = json.loads(resp.text)['run_id']
run_results_payload = {
   "run_id": run_id
}
 
run_incomplete = True
while run_incomplete:
    resp = requests.get('https://<url>/api/2.1/jobs/runs/get', json=run_results_payload, headers={'Authorization': 'Bearer <access-token>'})
    state = json.loads(resp.text)['state']['life_cycle_state']
    if state == 'TERMINATED':
        print("COMPLETED: {}".format(run_id))
        resp = requests.get('https://<url>/api/2.1/jobs/runs/get-output', json=run_results_payload, headers={'Authorization': 'Bearer <access_token>'})
        output = json.loads(resp.text)['notebook_output']
        print(output)
        run_incomplete = False
    else:
        time.sleep(2)

# Databricks Notebook function
 
def print_number(data=None):
  return data

Below are screenshots of the run and results in the UI

EDIT: I was able to obtain notebook results by adding dbutils.notebook.exit("return_value") to the notebook being executed via the API call, but still struggling to get the notebook to accept the base_parameters provided in the API call

Re: Databricks Jobs API not returning notebook run results?

apayne — Mon, 28 Nov 2022 19:30:59 GMT

Resolved this by using dbutils within the notebook being called from the API.

# databricks notebook function
 
data = dbutils.widgets.get('data') # pulls base_parameters from API call
 
def add_test(i):
    result = i + ' COMPLETE'
    return result
 
dbutils.notebook.exit(str(add_test(data))

Output from API call:

COMPLETED

{'result': 'test COMPLETE', 'truncated': False}

topic Re: Databricks Jobs API not returning notebook run results? in Data Engineering

Databricks Jobs API not returning notebook run results?

Re: Databricks Jobs API not returning notebook run results?