topic should have the option to mark succeeded with failures as a failure rather than a success in Data Engineering

should have the option to mark succeeded with failures as a failure rather than a success

kenmyers-8451 — Mon, 28 Jul 2025 14:00:52 GMT

Hi we are having an issue with the way succeeded with failures is handled. We will get emails telling us that we have a failure, which is correct, but then the pipeline actually treats it like a success and keeps going, but actually we would like to report the whole process as a failure. We have a pretty nested series of workflows that looks like this

- main workflow |- sub workflow |- sub sub workflow |- some failing task that isn't a leaf/terminating node

Now here's the same structure above but with notes about our issues with the state (this looked messy hence putting just the structure above):

- main workflow [email says "success", state in databricks says "succeeded", but this is wrong because sub workflow should really be classified as a failure, not a success] |- sub workflow [email says "success", state in databricks says "succeeded", but this is wrong because sub sub workflow should really be classified as a failure, not a success] |- sub sub workflow [email sent says "failure", state in databricks says "succeeded with failures", ideally this should probably just be a failure] |- some failing task that isn't a leaf/terminating node [marked as failure]

So in summary, it would be ideal to mark a workflow run that has "succeeded with faillures" as a failure rather than a succeess. I haven't found many people talking about this but did find someone on reddit talking about the same issue.

Re: should have the option to mark succeeded with failures as a failure rather than a success

Advika — Tue, 29 Jul 2025 14:50:29 GMT

Hello @kenmyers-8451!

It's a valid product feedback that would be worth flagging.
In the meantime, to ensure your pipeline run fails when any sub-task fails, you can add a final sentinel task at the end of your workflow. This task should programmatically inspect the state of all prior tasks. If it detects any task that failed or was only partially successful, it can raise an error, causing the overall workflow to fail and trigger accurate notifications.

Re: should have the option to mark succeeded with failures as a failure rather than a success

kenmyers-8451 — Tue, 05 Aug 2025 13:31:29 GMT

thanks @Advika we'll give that a shot for now

Re: should have the option to mark succeeded with failures as a failure rather than a success

Anish_2 — Sun, 22 Mar 2026 13:48:23 GMT

@Advika Currently, if we put

run_if: ALL_DONE on task level, it is flagged as succeededwithfailures. But there is another workflow under which this workflow runs. In master workflow, it is marked as succeeded only not succeededwithfailures.

My orchestration is done as below:

master workflow:

sub workflow:

task 1:

task 2:

Task 1 got failed and sub workflow is marked as succeededwithfailures but when i saw logs of master workflow, it is showing sub workflow as succeeded. Do you have any idea when this can be fixed?