Databricks Jobs API not returning notebook run results?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-23-2022 09:32 AM
Calling a databricks notebook using the Rest API, can confirm that it is executing the notebook, but is not accepting my parameters or returning a notebook output. Any ideas on what I am doing wrong here?
My code and notebook function are below, trying to pass a string to the notebook function and have it return the results.
# Databricks Rest API calls from IDE
import requests
import json
import time
job_payload = {
"run_name": 'execute_test',
"existing_cluster_id": '<cluster-id>',
"notebook_task":
{
"notebook_path": '/Users/<user>/testing',
"source": "WORKSPACE",
"base_parameters": {"data": "test"}
}
}
resp = requests.post('https://<url>/api/2.1/jobs/runs/submit', json=job_payload, headers={'Authorization': 'Bearer <access token>'})
run_id = json.loads(resp.text)['run_id']
run_results_payload = {
"run_id": run_id
}
run_incomplete = True
while run_incomplete:
resp = requests.get('https://<url>/api/2.1/jobs/runs/get', json=run_results_payload, headers={'Authorization': 'Bearer <access-token>'})
state = json.loads(resp.text)['state']['life_cycle_state']
if state == 'TERMINATED':
print("COMPLETED: {}".format(run_id))
resp = requests.get('https://<url>/api/2.1/jobs/runs/get-output', json=run_results_payload, headers={'Authorization': 'Bearer <access_token>'})
output = json.loads(resp.text)['notebook_output']
print(output)
run_incomplete = False
else:
time.sleep(2)# Databricks Notebook function
def print_number(data=None):
return dataBelow are screenshots of the run and results in the UI
EDIT: I was able to obtain notebook results by adding dbutils.notebook.exit("return_value") to the notebook being executed via the API call, but still struggling to get the notebook to accept the base_parameters provided in the API call
- Labels:
-
Databricks notebook
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-28-2022 11:30 AM
Resolved this by using dbutils within the notebook being called from the API.
# databricks notebook function
data = dbutils.widgets.get('data') # pulls base_parameters from API call
def add_test(i):
result = i + ' COMPLETE'
return result
dbutils.notebook.exit(str(add_test(data))Output from API call:
COMPLETED
{'result': 'test COMPLETE', 'truncated': False}