Make the job fail if a task fail

[tramtran]

Hi everyone,

I have a job with 2 tasks running independently. If one of them fails, the remaining task continues to run. I would like the job to fail if any task fails.

Is there any way to do that?

Thank you!

[holly]

Can you share why you want that to happen? If it's a case of saving on compute, you can repair the run and only rerun the parts that failed.

If you want the status code to be a failure, you could use the if/else logic with both tasks as dependents to then point to a failure script to 'fail' the final step.

[tramtran]

Actually, my job has multiple streaming tasks running independently. I would like to make the whole job stop if one of the streaming tasks fails.

For example, one of the tasks failed after 8 minutes of execution, but the others continued running for 35 hours until I found and canceled them. I would the job to fail if any task fails.

[holly]

Unfortunately, that's not something that can be done today. It's a good idea, so I'll raise it with the product team.

As for saving you 35 hours, you can set up alerting should a single task fail. The alerts can be emails, but can also be slack/teams messages or anything with a webhook. Setup docs are here: https://docs.databricks.com/en/workflows/jobs/job-notifications.html

[mhiltner]

An idea, not sure if it helps you... You could set up a new task dependent on every other task, but with the run option set to "Run If at least one dependency failed". This new task would be a simple API call to the CANCEL RUN or CANCEL ALL RUNS OF A JOB endpoint. -- https://docs.databricks.com/api/workspace/jobs/cancelallruns

[holly]

The problem with that is that you're dependent on the other task 'finishing' and if it's an always on streaming job, by definition it won't finish

[Edthehead]

Extending to what @mhiltner has suggested, let's  say you have 2 streaming tasks streamA and streamB. Create 2 separate tasks taskA and taskB. Each of these tasks should execute the same notebook which makes an API call to the CANCEL RUN or CANCEL ALL RUNS OF A JOB endpoint.

Task settings should be as follows :

1. taskA be dependent on streamA with Run if dependencies as All failed. So taskA will only run when streamA fails.
2. taskB to be dependent on streamB with Run if dependencies as All failed. So taskB will only run when streamB fails.

So now, if streamA fails, taskA will run and cancel the run. If streamB fails, taskB will run and cancel the run.