topic Re: Executing Notebooks - Run All Cells vs Run All Below in Data Engineering

Executing Notebooks - Run All Cells vs Run All Below

lei_armstrong — Tue, 19 Oct 2021 12:57:41 GMT

Due to dependencies, if one of our cells errors then we want the notebook to stop executing.

We've noticed some odd behaviour when executing notebooks depending on if "Run all cells in this notebook" is selected from the header versus "Run All Below".

In the example code below I've added an extra bracket to force the python to fail and raise an exception. If I run the notebook using "Run all cells in this notebook" then the exception is correctly thrown and the rest of the cells/commands in the notebook are skipped.

However, if I use "Run All Below" then all cells are executed regardless of any exceptions or failures. We're tried using dbutils.notebook.exit but it doesn't work and subsequent cells are still run.

Is this the intended behaviour? It's frustrating when trying to rerun just part of a notebook.

Example Code:

Cmd 1

%python

try:

spark.sql("""SELECT NOW()) AS force_error""")

except:

print("Error running SQL")

raise Exception("Error running SQL")

Cmd 2

SELECT NOW())

Cmd 3

SELECT NOW()

Re: Executing Notebooks - Run All Cells vs Run All Below

-werners- — Tue, 19 Oct 2021 15:11:24 GMT

That is how it works. When you do 'run all', the cells after the error are skipped.

When you do run all below, they all get executed (and mostly throw an error).

Databricks probably has a reason for this behavior. The notebook experience is not bad but certainly not perfect.

Re: Executing Notebooks - Run All Cells vs Run All Below

Dan_Z — Tue, 19 Oct 2021 15:50:25 GMT

Yes- this is expected behavior. Usually customers want all the cells to run regardless if one failed in between. Usually future cells will error our if they are dependent anyway. If you can suggest a feature that will provide that functionality I can make a feature request on your behalf.

Re: Executing Notebooks - Run All Cells vs Run All Below

lei_armstrong — Wed, 20 Oct 2021 10:12:29 GMT

Thanks all for your responses, just seems strange the behaviour differs and isn't documented publicly (that I can find). Ideally it would be configurable as part of the notebook settings. I've raised a new feature request here : Notebook Settings for handling errors · Community (azure.com)

Re: Executing Notebooks - Run All Cells vs Run All Below

Dan_Z — Fri, 22 Oct 2021 14:52:50 GMT

@Laura McGill , this question just came up again here:

https://community.databricks.com/s/question/0D53f00001PonToCAJ/executing-notebooks-run-all-cells-vs-run-all-below

I went ahead and created an internal feature request. You can refer to it with DB-I-4250. If you'd like to help prioritize it, please contact your Account Executive or Customer Success Engineer and tell them your company needs DB-I-4250. Being able to attach your company name to feature requests helps the product team prioritize them.

Cheers!

Re: Executing Notebooks - Run All Cells vs Run All Below

pinecone — Mon, 15 Aug 2022 12:06:22 GMT

I second this request. It's odd that the behaviour is different when running all vs. running all below. Please make it consistent and document properly.

Re: Executing Notebooks - Run All Cells vs Run All Below

sukanya09 — Thu, 25 Jul 2024 16:07:32 GMT

Has this been implemented? I have created a job using notebook. My notebook has 6 cells and if the code in first cell fails it should not run the rest of the cells