Re: Orchestrate run of a folder

cmilligan · ‎11-03-2022

I'm needing to run the contents of a folder, which can change over time. Is there a way to set up a notebook that can orchestrate running all notebooks in a folder? My though was if I could retrieve a list of the notebooks I could create a loop to run them

weldermartins · ‎11-03-2022

hello, could you detail what the file type would be? To run several files you could enter the *(asterisk)

Hubert-Dudek · ‎11-03-2022

List all notebooks by making API call and then run them by using dbutils.notebook.run:

import requests
ctx = dbutils.notebook.entry_point.getDbutils().notebook().getContext()
host_name = ctx.tags().get("browserHostName").get()
host_token = ctx.apiToken().get()
 
notebook_folder = '/Users/hubert.dudek@databrickster.com'
 
response = requests.get(
f'https://{host_name}/api/2.0/workspace/list',
headers={'Authorization': f'Bearer {host_token}'},
json={'path': notebook_folder}
).json()
 
for notebook in response['objects']:
    if notebook['object_type'] == 'NOTEBOOK':
        dbutils.notebook.run(notebook['path'], 1800)

My blog: https://databrickster.medium.com/

View solution in original post

cmilligan · ‎11-03-2022

So that seemed to work when running the notebook manually. Unfortunately when I set it as a job to run it throws an error. Also will this run them sequentially? I need them to run one at a time in order.