cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Running a notebook as 'Run all below' when sheduled via Azure DataFactory

LeenB
New Contributor

We have a notebook with a lot of subsequent cells that can run independent from each other. When we execute the notebook manually via 'Run all', the runs stops when an error is thrown. When we execute manually via 'Run all below', the run proceeds till the end, even if there was an error. If I understand correctly, this is how it works in DataBricks.
We have scheduled the run of this notebook now on a daily basis as an Activity (Databrick Notebook) in Azure DataFactory. The problem for us is that the execution is always running as 'Run all' i.e. the run stops when an error is given. Actually we want that the run proceeds until the end of the notebook, also when there is an error (so the run with 'Run all below'). Is there a way to do this via DataFactory or to add/change something in the notebook/cluster, that it always runs as 'Run all below'?

1 REPLY 1

PiotrMi
Contributor

Hi @LeenB 

For example each cell execution you can build up with try except command. Example below

try: 

    print("Hello world") 

   #your code of each cell

except Exception as e:

    print("Issue with printing hello world")

For sure it is not recommended approach but it will work

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now