topic Re: How to exit the entire job in the orchestration scenario? in Data Engineering

How to exit the entire job in the orchestration scenario?

QPeiran — Thu, 11 Aug 2022 01:01:21 GMT

Hi, can anybody answer this question I posted on StackOverflow?

https://stackoverflow.com/questions/73314048/databricks-how-to-exit-the-entire-job-in-the-notebooks-orchestration-scenario

Re: How to exit the entire job in the orchestration scenario?

Vidula — Thu, 08 Sep 2022 11:36:15 GMT

Hi there @Peiran Quan

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!

Re: How to exit the entire job in the orchestration scenario?

CarterM — Mon, 10 Apr 2023 23:12:02 GMT

@Vidula Khanna @Vidula Khanna

We are experiencing the same issue in our Workflows and I was wondering if there has been any update.

We need the functionality to call a method similar to `dbutils.notebook.exit` in a notebook that will cancel the execution of all the following dependent Tasks or the Job as a whole, but this does not seem to exist in Databricks.

For example,

In the following Example Job, if a certain criteria is met in the Task_1 notebook then I want the Example Job to Stop without firing an error. The idea is that we wouldn't have to waste any unnecessary execution in Task_2 and Task_3.

Is there a way to make a feature request or are there plans to implement this functionality already?

Re: How to exit the entire job in the orchestration scenario?

Debayan — Thu, 11 Aug 2022 14:43:41 GMT

As of now there is no way to stop the present cell after a successful run in "run all" scenario.

Additionally, you can always put an error in the cell where you want run all to stop since errors stop "run all".

Also, you can run all cells above and run all cells below. For example: If you want to run all cells up to cell X, go to cell X+1 and “run all cells above”.