topic Re: dbutils.notebook.run() getting general error instead specific in Data Engineering

dbutils.notebook.run() getting general error instead specific

Itai_Sharon — Sun, 03 Aug 2025 10:59:41 GMT

Hi,

In a python file using dbutils.notebook.run() I'm running specific notebook.
The notebook is failling but i'm getting a general error log instead the real specific log.
When I'm running the notebook directly - I'm getting the specific error log.

general log:
Error: An error occurred while calling o560.run. : com.databricks.WorkflowException: com.databricks.NotebookExecutionException: FAILED: Workload failed, see run output for details at com.databricks.workflow.WorkflowDriver.run(WorkflowDriver.scala:99) at com.databricks.dbutils_v1.impl.NotebookUtilsImpl.run(NotebookUtilsImpl.scala:130) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397) at py4j.Gateway.invoke(Gateway.java:306) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:199) at py4j.ClientServerConnection.run(ClientServerConnection.java:119) at java.lang.Thread.run(Thread.java:750) Caused by: com.databricks.NotebookExecutionException: FAILED: Workload failed, see run output for details at com.databricks.workflow.WorkflowDriver.run0(WorkflowDriver.scala:147) at com.databricks.workflow.WorkflowDriver.run(WorkflowDriver.scala:94) ... 13 more

real specific log:
FileNotFoundError: [Errno 2] No such file or directory: '/Workspace/Varonis/ETL-Processing/packages/membership/src/tables/silver/memberships_all_paths/table.py'

Re: dbutils.notebook.run() getting general error instead specific

Vinay_M_R — Mon, 04 Aug 2025 05:16:26 GMT

Hi @Itai_Sharon,

When you call a Databricks notebook from another notebook (or from a Python file) using dbutils.notebook.run(), if the child notebook fails, the default behavior is for the error in the child to be wrapped and propagated up as a generic WorkflowException with a general message like: com.databricks.NotebookExecutionException: FAILED: Workload failed, see run output for details.

This means that the original exception (e.g., FileNotFoundError) and its stacktrace are not automatically surfaced to the caller—only the general workflow failure message appears.

If you run the same notebook manually, you do see the full, specific Python error log with stacktrace in the notebook UI.

The error thrown by dbutils.notebook.run() is always a WorkflowException, which wraps the real error from the child notebook.

There is no automatic surfacing of the nested error's stacktrace or details in the value returned to the caller.

I am sharing official documentation for your reference:https://learn.microsoft.com/en-us/azure/databricks/notebooks/notebook-workflows#handle-errors

Re: dbutils.notebook.run() getting general error instead specific

Itai_Sharon — Mon, 04 Aug 2025 06:28:39 GMT

Thanks @Vinay_M_R .

Are there any workaround options available?

I noticed that one approach involves modifying the child's notebook to include something like:

try: # your code except Exception as e: dbutils.notebook.exit(str(e))

However, I’d prefer not to change the notebook I’m calling.
Are there any other ways to capture the specific error without modifying the child notebook?

Re: dbutils.notebook.run() getting general error instead specific

Itai_Sharon — Tue, 05 Aug 2025 11:41:17 GMT

@Vinay_M_R

BTW, when trying to run a job using Databricks API, I encounter the same issue (general "FAILED: Workload failed"):

from databricks.sdk import WorkspaceClient client = WorkspaceClient() run = client.jobs.run_now(job_id)

error message:

state_message='Workload failed, see run output for details'