dbutils.notebook.run() getting general error inste...

Itai_Sharon · ‎08-03-2025

Hi,

In a python file using dbutils.notebook.run() I'm running specific notebook.
The notebook is failling but i'm getting a general error log instead the real specific log.
When I'm running the notebook directly - I'm getting the specific error log.

general log:
Error: An error occurred while calling o560.run. : com.databricks.WorkflowException: com.databricks.NotebookExecutionException: FAILED: Workload failed, see run output for details at com.databricks.workflow.WorkflowDriver.run(WorkflowDriver.scala:99) at com.databricks.dbutils_v1.impl.NotebookUtilsImpl.run(NotebookUtilsImpl.scala:130) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:397) at py4j.Gateway.invoke(Gateway.java:306) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:199) at py4j.ClientServerConnection.run(ClientServerConnection.java:119) at java.lang.Thread.run(Thread.java:750) Caused by: com.databricks.NotebookExecutionException: FAILED: Workload failed, see run output for details at com.databricks.workflow.WorkflowDriver.run0(WorkflowDriver.scala:147) at com.databricks.workflow.WorkflowDriver.run(WorkflowDriver.scala:94) ... 13 more

real specific log:
FileNotFoundError: [Errno 2] No such file or directory: '/Workspace/Varonis/ETL-Processing/packages/membership/src/tables/silver/memberships_all_paths/table.py'

Vinay_M_R · ‎08-03-2025

Hi @Itai_Sharon,

When you call a Databricks notebook from another notebook (or from a Python file) using dbutils.notebook.run(), if the child notebook fails, the default behavior is for the error in the child to be wrapped and propagated up as a generic WorkflowException with a general message like: com.databricks.NotebookExecutionException: FAILED: Workload failed, see run output for details.

This means that the original exception (e.g., FileNotFoundError) and its stacktrace are not automatically surfaced to the caller—only the general workflow failure message appears.

If you run the same notebook manually, you do see the full, specific Python error log with stacktrace in the notebook UI.

The error thrown by dbutils.notebook.run() is always a WorkflowException, which wraps the real error from the child notebook.

There is no automatic surfacing of the nested error's stacktrace or details in the value returned to the caller.

I am sharing official documentation for your reference:https://learn.microsoft.com/en-us/azure/databricks/notebooks/notebook-workflows#handle-errors

Itai_Sharon · ‎08-03-2025

Thanks @Vinay_M_R .

Are there any workaround options available?

I noticed that one approach involves modifying the child's notebook to include something like:

try:
    # your code
except Exception as e:
    dbutils.notebook.exit(str(e))

However, I’d prefer not to change the notebook I’m calling.
Are there any other ways to capture the specific error without modifying the child notebook?

Itai_Sharon · ‎08-05-2025

@Vinay_M_R

BTW, when trying to run a job using Databricks API, I encounter the same issue (general "FAILED: Workload failed"):

from databricks.sdk import WorkspaceClient

client = WorkspaceClient()
run = client.jobs.run_now(job_id)

error message:

state_message='Workload failed, see run output for details'

dbutils.notebook.run() getting general error instead specific