Catch when a notebook fails and terminate command in threaded parallel notebook run

cmilligan — Sat, 05 Nov 2022 19:53:52 GMT

I have a command that is running notebooks in parallel using threading. I want the command to fail whenever one of the notebooks that is running fails. Right now it is just continuing to run the command.

Below is the command line that I'm currently running:

q = Queue()
worker_count = 3 
 
def run_notebook(notebook):
  print(notebook)
  dbutils.notebook.run(notebook, 30, {"begin_date": begin_date, "end_date": end_date})
 
def run_tasks(function, q):
    while not q.empty():
      value = q.get()
      function(value)
      q.task_done()
 
for x in range(len(notebook_steps)): 
  for x in notebook_steps[x]:
    q.put(x)
      
  for i in range(worker_count):
    t=Thread(target=run_tasks, args=(run_notebook, q))
    t.daemon = False
    t.start()
 
  q.join()

topic Catch when a notebook fails and terminate command in threaded parallel notebook run in Data Engineering

Catch when a notebook fails and terminate command in threaded parallel notebook run