dbutils.notebooks.exit() is not returning results to the variable only on some notebooks

[v_n66]

dbutils.notebooks.exit() is not returning results to the variable only on some notebooks

issue is Parent child notebook communication and need some solution

[SteveW]

We believe that we're experiencing the same issue, and that it is affecting other users too.

result = dbutils.notebook.run(notebook_to_run, 0, notebook_args)
print(result)

We can see that the run of notebook_to_run completed successfully with (we believe) an exit message, however the print line is showing "none". When we re-run the relevant workflow the issue resolves, however the issue the occurs in the next step which has similar code, re-running again resolves that step so we have had to run the workflow once per step in order for Databricks to recognise that the steps have succeeded.

[v_n66]

@SteveW , You are correct, we are also facing the same issue from yesterday.

Scenario 1:
The issue only occurs with notebooks that use structured streaming. For non-streaming notebooks, dbutils.notebook.exit() works as expected.

Scenario 2:
The failure happens only when there's actual data to be streamed from Bronze to Silver and Gold layers. On the initial run — when data is present — the process fails. However, on a retry (when there's no new data to stream), it succeeds without any issues.

Scenario 3:
We added .awaitTermination() to all streaming queries, and after that change, the job started succeeding — at least on retries.

We suspect the issue is related to the Databricks IDE/UI behavior, and here's what we observed:

A notebook run at 06:25 AM IST on May 9th did not display the streaming dashboard UI, and we haven't got any issue in the parent Child Handshake using dbutils.notebook.exit(json) command

however, with the same code, we could see that the streaming window popped for 05.25AM run and this time child notebook, doesn't send the dbutils.notebook.exit(json) to the parent.

[SteveW]

Thanks for that extra info @v_n66 - very useful!

[lorenzoscandola]

I'm facing the same problem, I hope someone will solve it next monday 😵

[SteveW]

Update: as of Sunday morning NZT we haven't seen the workflow failures that we were, so it may be that the issue is resolved. Are you seeing the same @v_n66 and @lorenzoscandola ?

[SteveW · Accepted solution]

From the support ticket we logged (via MS): "The Databricks engineering team identified a bug related to dbutils.notebook.exit /dbutils.notebook.run in Jobs API. The Jobs engineering team rolled back all environments to the previously working version of the Jobs API." ... and from less official channels: "there was a rollback to the Jobs API sometime on Saturday evening by Databricks."

[lorenzoscandola]

@SteveW atm everything works! 

[v_n66]

@SteveW @lorenzoscandola Its working without any issue from Yesterday. Thank you guys for the support.