topic Re: dbutils.notebooks.exit() is not returning results to the variable only on some notebooks in Data Engineering

dbutils.notebooks.exit() is not returning results to the variable only on some notebooks

v_n66 — Thu, 08 May 2025 14:40:39 GMT

dbutils.notebooks.exit() is not returning results to the variable only on some notebooks

issue is Parent child notebook communication and need some solution

Re: dbutils.notebooks.exit() is not returning results to the variable only on some notebooks

SteveW — Fri, 09 May 2025 00:51:22 GMT

We believe that we're experiencing the same issue, and that it is affecting other users too.

result = dbutils.notebook.run(notebook_to_run, 0, notebook_args)
print(result)

We can see that the run of notebook_to_run completed successfully with (we believe) an exit message, however the print line is showing "none". When we re-run the relevant workflow the issue resolves, however the issue the occurs in the next step which has similar code, re-running again resolves that step so we have had to run the workflow once per step in order for Databricks to recognise that the steps have succeeded.

Re: dbutils.notebooks.exit() is not returning results to the variable only on some notebooks

v_n66 — Fri, 09 May 2025 02:34:59 GMT

@SteveW , You are correct, we are also facing the same issue from yesterday.

Scenario 1:
The issue only occurs with notebooks that use structured streaming. For non-streaming notebooks, dbutils.notebook.exit() works as expected.

Scenario 2:
The failure happens only when there's actual data to be streamed from Bronze to Silver and Gold layers. On the initial run — when data is present — the process fails. However, on a retry (when there's no new data to stream), it succeeds without any issues.

Scenario 3:
We added .awaitTermination() to all streaming queries, and after that change, the job started succeeding — at least on retries.

We suspect the issue is related to the Databricks IDE/UI behavior, and here's what we observed:

A notebook run at 06:25 AM IST on May 9th did not display the streaming dashboard UI, and we haven't got any issue in the parent Child Handshake using dbutils.notebook.exit(json) command

however, with the same code, we could see that the streaming window popped for 05.25AM run and this time child notebook, doesn't send the dbutils.notebook.exit(json) to the parent.

Re: dbutils.notebooks.exit() is not returning results to the variable only on some notebooks

SteveW — Fri, 09 May 2025 02:39:36 GMT

Thanks for that extra info @v_n66 - very useful!

Re: dbutils.notebooks.exit() is not returning results to the variable only on some notebooks

lorenzoscandola — Fri, 09 May 2025 15:35:55 GMT

I'm facing the same problem, I hope someone will solve it next monday 😵

Re: dbutils.notebooks.exit() is not returning results to the variable only on some notebooks

SteveW — Sun, 11 May 2025 21:13:53 GMT

Update: as of Sunday morning NZT we haven't seen the workflow failures that we were, so it may be that the issue is resolved. Are you seeing the same @v_n66 and @lorenzoscandola ?

Re: dbutils.notebooks.exit() is not returning results to the variable only on some notebooks

SteveW — Mon, 12 May 2025 05:32:19 GMT

From the support ticket we logged (via MS): "The Databricks engineering team identified a bug related to dbutils.notebook.exit /dbutils.notebook.run in Jobs API. The Jobs engineering team rolled back all environments to the previously working version of the Jobs API." ... and from less official channels: "there was a rollback to the Jobs API sometime on Saturday evening by Databricks."

Re: dbutils.notebooks.exit() is not returning results to the variable only on some notebooks

lorenzoscandola — Mon, 12 May 2025 09:54:30 GMT

@SteveW atm everything works!

Re: dbutils.notebooks.exit() is not returning results to the variable only on some notebooks

v_n66 — Mon, 12 May 2025 12:34:09 GMT

@SteveW @lorenzoscandola Its working without any issue from Yesterday. Thank you guys for the support.