topic Re: Make the job fail if a task fail in Data Engineering

Make the job fail if a task fail

tramtran — Fri, 02 Aug 2024 06:11:23 GMT

Hi everyone,

I have a job with 2 tasks running independently. If one of them fails, the remaining task continues to run. I would like the job to fail if any task fails.

Is there any way to do that?

Thank you!

Re: Make the job fail if a task fail

holly — Fri, 02 Aug 2024 07:15:32 GMT

Can you share why you want that to happen? If it's a case of saving on compute, you can repair the run and only rerun the parts that failed.

If you want the status code to be a failure, you could use the if/else logic with both tasks as dependents to then point to a failure script to 'fail' the final step.

Re: Make the job fail if a task fail

tramtran — Fri, 02 Aug 2024 08:27:51 GMT

Actually, my job has multiple streaming tasks running independently. I would like to make the whole job stop if one of the streaming tasks fails.

For example, one of the tasks failed after 8 minutes of execution, but the others continued running for 35 hours until I found and canceled them. I would the job to fail if any task fails.

Re: Make the job fail if a task fail

holly — Fri, 02 Aug 2024 13:30:00 GMT

Unfortunately, that's not something that can be done today. It's a good idea, so I'll raise it with the product team.

As for saving you 35 hours, you can set up alerting should a single task fail. The alerts can be emails, but can also be slack/teams messages or anything with a webhook. Setup docs are here: https://docs.databricks.com/en/workflows/jobs/job-notifications.html

Re: Make the job fail if a task fail

mhiltner — Sat, 03 Aug 2024 18:45:50 GMT

An idea, not sure if it helps you... You could set up a new task dependent on every other task, but with the run option set to "Run If at least one dependency failed". This new task would be a simple API call to the CANCEL RUN or CANCEL ALL RUNS OF A JOB endpoint. -- https://docs.databricks.com/api/workspace/jobs/cancelallruns

Re: Make the job fail if a task fail

holly — Mon, 05 Aug 2024 15:15:54 GMT

The problem with that is that you're dependent on the other task 'finishing' and if it's an always on streaming job, by definition it won't finish

Re: Make the job fail if a task fail

Edthehead — Tue, 06 Aug 2024 07:52:05 GMT

Extending to what @mhiltner has suggested, let's say you have 2 streaming tasks streamA and streamB. Create 2 separate tasks taskA and taskB. Each of these tasks should execute the same notebook which makes an API call to the CANCEL RUN or CANCEL ALL RUNS OF A JOB endpoint.

Task settings should be as follows :

taskA be dependent on streamA with Run if dependencies as All failed. So taskA will only run when streamA fails.
taskB to be dependent on streamB with Run if dependencies as All failed. So taskB will only run when streamB fails.

So now, if streamA fails, taskA will run and cancel the run. If streamB fails, taskB will run and cancel the run.