topic Re: Gracefully stop a job based on condition in Data Engineering

Gracefully stop a job based on condition

erigaud — Fri, 14 Jul 2023 06:26:02 GMT

Hello,

I have a job with many tasks running on a schedule, and the first tasks checks a condition. Based on the condition, I would either want to continue the job as normal, or to stop right away and don't run all the other tasks. Is there a way to do this without having the job in a failed state ?

Re: Gracefully stop a job based on condition

menotron — Fri, 14 Jul 2023 12:51:13 GMT

You can use the dbutils.notebook.exit("returnValue") as explained here stop execution of a notebook gracefully.

if condition: pass else: dbutils.notebook.exit("Aborting as ondition not met. Further tasks will be skipped")

Re: Gracefully stop a job based on condition

pvignesh92 — Fri, 14 Jul 2023 13:14:18 GMT

@erigaud Hi. The below two approaches could help.

1. dbutils.notebook.exit() --> This will stop the job. You can even pass any values in the parenthesis to print based on your requirement.

2. Using sys.exit(0) -> This comes with sys module and you can use this as well to exit your job. Both will work. You can try and let me know.

Re: Gracefully stop a job based on condition

erigaud — Fri, 14 Jul 2023 13:33:15 GMT

I would want the full job to stop (including subsequent tasks) but running dbutils.notebook.exit() will success the current task and move on to the next task right ? I would want the job to finish right away in a success state

Re: Gracefully stop a job based on condition

menotron — Fri, 14 Jul 2023 14:35:02 GMT

Yes, you're correct. The downstream tasks would still be triggered.
You can try cancelling the run using the Jobs API? But I don't think that is what you are looking for as you want the job run to have success state.

Maybe propagating the check using Task values can be an option or simply add the check to every task.

Re: Gracefully stop a job based on condition

erigaud — Fri, 14 Jul 2023 14:45:47 GMT

I think the best way to accomplish this would be to either propagate the check, as mentionned by @menotron, or have the initial task in another job, and only run the second job if the condition is met. Obviously it depends on the use case. Thank you for taking the time to answer !