Running a notebook as 'Run all below' when sheduled via Azure DataFactory

LeenB — Thu, 27 Feb 2025 10:16:23 GMT

We have a notebook with a lot of subsequent cells that can run independent from each other. When we execute the notebook manually via 'Run all', the runs stops when an error is thrown. When we execute manually via 'Run all below', the run proceeds till the end, even if there was an error. If I understand correctly, this is how it works in DataBricks.
We have scheduled the run of this notebook now on a daily basis as an Activity (Databrick Notebook) in Azure DataFactory. The problem for us is that the execution is always running as 'Run all' i.e. the run stops when an error is given. Actually we want that the run proceeds until the end of the notebook, also when there is an error (so the run with 'Run all below'). Is there a way to do this via DataFactory or to add/change something in the notebook/cluster, that it always runs as 'Run all below'?

Re: Running a notebook as 'Run all below' when sheduled via Azure DataFactory

PiotrMi — Thu, 27 Feb 2025 11:30:06 GMT

Hi @LeenB

For example each cell execution you can build up with try except command. Example below

try:

print("Hello world")

#your code of each cell

except Exception as e:

print("Issue with printing hello world")

For sure it is not recommended approach but it will work

topic Running a notebook as 'Run all below' when sheduled via Azure DataFactory in Data Engineering

Running a notebook as 'Run all below' when sheduled via Azure DataFactory

Re: Running a notebook as 'Run all below' when sheduled via Azure DataFactory