topic Re: Orchestrate run of a folder in Data Engineering

Orchestrate run of a folder

cmilligan — Thu, 03 Nov 2022 12:36:40 GMT

I'm needing to run the contents of a folder, which can change over time. Is there a way to set up a notebook that can orchestrate running all notebooks in a folder? My though was if I could retrieve a list of the notebooks I could create a loop to run them

Re: Orchestrate run of a folder

weldermartins — Thu, 03 Nov 2022 12:49:43 GMT

hello, could you detail what the file type would be? To run several files you could enter the *(asterisk)

Re: Orchestrate run of a folder

Hubert-Dudek — Thu, 03 Nov 2022 14:26:34 GMT

List all notebooks by making API call and then run them by using dbutils.notebook.run:

import requests
ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
host_name = ctx.tags().get("browserHostName").get()
host_token = ctx.apiToken().get()
 
notebook_folder = '/Users/hubert.dudek@databrickster.com'
 
response = requests.get(
f'https://{host_name}/api/2.0/workspace/list',
headers={'Authorization': f'Bearer {host_token}'},
json={'path': notebook_folder}
).json()
 
for notebook in response['objects']:
    if notebook['object_type'] == 'NOTEBOOK':
        dbutils.notebook.run(notebook['path'], 1800)

Re: Orchestrate run of a folder

cmilligan — Thu, 03 Nov 2022 19:17:02 GMT

So that seemed to work when running the notebook manually. Unfortunately when I set it as a job to run it throws an error. Also will this run them sequentially? I need them to run one at a time in order.