Databricks Workflow Task Failure - Custom Error Messages

[ChristianRRL]

Hi there, kind of silly question but I'm hoping there's a simple way to address this.

I'm trying to make sure that when certain job failure conditions arise, that a workflow task step fails. I've tried this with both raise RuntimeError and sys.exit, but in both cases I'm seeing more in the output than I would like.

I would like to output the most minimal amount of text so that users who are troubleshooting can easily read the custom error message we are outputing.

Is there another/better approach to what I'm trying to do, or is this a Databricks limitation in how raised errors or system exit messages are output?

Also, would one be recommended over the other (raised exception vs system exit)?

Below are examples for both:

[K_Anudeep]

Hey @ChristianRRL !

When a Python wheel/notebook/script task fails, Jobs always does three things:

1. Render the Python traceback in task output (IPython / Databricks REPL showtraceback).
2. Extract a one-line summary (RuntimeError: <your message>), which is the yellow line at the bottom of your screenshot.
3. Set the task termination message to Workload failed, see run output for details. That string is a fixed platform copy for “user code failed.” You cannot replace it with your custom message.

Python wheel tasks are executed as notebook commands, so they receive the same wrapping. Jobs already strips some internal wheel frames, but your main() / raise frames stay.

There is no supported API today to fail a task with only a custom string and no traceback (dbutils.notebook.exit(...)). Also, it does not fail the task; it succeeds and returns a value.

[anuj_lathi]

This is mostly how notebook tasks surface errors, not something you can fully control from your code. In a notebook task, sys.exit() raises SystemExit, which IPython intercepts. That's where the "An exception has occurred, use %tb…" line and the To exit: use 'exit', 'quit'… warning come from. I'd use raise over sys.exit, and if you want cleaner output, move the check out of a notebook task.

Raise vs. sys.exit

• Raise an exception. This is the right way to fail a notebook task. The task fails and the message shows in the run output. A traceback comes with it, but it's the normal, expected output.
• Avoid sys.exit() in notebooks. It still fails the task, but you get the IPython wrapper text and the warning shown in your screenshot, so it's strictly noisier.
• Don't use dbutils.notebook.exit() to signal failure. It ends the run successfully, so downstream tasks will still run.

Getting the output down

1. Raise a short, purpose-built exception with the message you want users to read:

class ContentGateError(Exception):
    pass

if not passed:
    raise ContentGateError("run_content_gate failed: 1,234 rows missing required content_id")

The last line of the error will be ContentGateError: run_content_gate failed: …, which is easy to spot. If you want the failure details to be the first thing people see, print a clear summary just before raising.

2. Keep the traceback short. Raise the error at the top level of the notebook cell rather than from deep inside helper functions, since each frame adds to the traceback. If you wrap logic in functions, return a pass/fail result and raise in one place at the end. Setting sys.tracebacklimit = 0 may trim the stack further, but that's an interpreter setting and IPython doesn't always honor it, so test it on your runtime before you rely on it.

3. Use a different task type if you want truly minimal output. The notebook UI adds the wrapper, so if the gate is just a check, there are two cleaner options [1]:

• Python script task. There's no IPython layer, so an uncaught exception or a non-zero exit gives plain Python-style output without the notebook warnings.
• SQL task or SQL-based check. If the gate can be expressed in SQL, raise_error or assert_true fails with just your message, in the form [USER_RAISED_EXCEPTION] <your message> SQLSTATE: P0001 [2]:

SELECT assert_true(
  (SELECT count(*) FROM main.my_schema.my_table WHERE content_id IS NULL) = 0,
  'run_content_gate failed: rows with NULL content_id found'
);

4. Consider a Condition (if/else) task for expected outcomes. If these failures are expected validation results rather than errors, compute the result in one task and branch on it. That lets you route to a notification or cleanup task instead of failing a run with a stack trace [1].

Recommendation

If you need to stay in a notebook, raise a small custom exception with a one-line message from the top level of a cell. If you want output that's actually minimal, make the gate a Python script task, or a SQL task with raise_error/assert_true.

References

[1] Lakeflow Jobs | Databricks on AWS — https://docs.databricks.com/aws/en/jobs
[2] Error handling in Databricks | Databricks on AWS — https://docs.databricks.com/aws/en/error-messages

[aayush_410]

It's a platform limitation. Notebook tasks run through IPython, which always prints a traceback for an unhandled exception, so you can't fully suppress the extra output. sys.exit doesn't help either. In a notebook it raises SystemExit, which gets rendered as an error too, and it's less clear to read.

Recommendation: raise an exception, not sys.exit.

python
class DataQualityError(Exception):
pass

if bad_condition:
raise DataQualityError("Row count check failed: expected > 0, got 0")
The task is marked failed, and your message appears on the last line of the output, which is where people look first.
A specific exception name (not a generic RuntimeError) makes the cause obvious at a glance.
Keep the message to a single line, with the check name and the values.

To get truly minimal output, use a Python script task (or wheel) instead of a notebook. There, sys.exit("Row count check failed: expected > 0, got 0") prints only that message to stderr and fails the task with exit code 1, with no traceback.

Other things that help:

Don't use dbutils.notebook.exit() to signal failure. It ends the task as successful.
To surface the reason without opening the run, set a task value (dbutils.jobs.taskValues.set(...)) and reference it in a downstream notification or condition task.

If you paste your two examples, I'll tighten them.