@SteveW , You are correct, we are also facing the same issue from yesterday.
Scenario 1:
The issue only occurs with notebooks that use structured streaming. For non-streaming notebooks, dbutils.notebook.exit() works as expected.
The issue only occurs with notebooks that use structured streaming. For non-streaming notebooks, dbutils.notebook.exit() works as expected.
Scenario 2:
The failure happens only when there's actual data to be streamed from Bronze to Silver and Gold layers. On the initial run — when data is present — the process fails. However, on a retry (when there's no new data to stream), it succeeds without any issues.
The failure happens only when there's actual data to be streamed from Bronze to Silver and Gold layers. On the initial run — when data is present — the process fails. However, on a retry (when there's no new data to stream), it succeeds without any issues.
Scenario 3:
We added .awaitTermination() to all streaming queries, and after that change, the job started succeeding — at least on retries.
We added .awaitTermination() to all streaming queries, and after that change, the job started succeeding — at least on retries.
We suspect the issue is related to the Databricks IDE/UI behavior, and here's what we observed:
A notebook run at 06:25 AM IST on May 9th did not display the streaming dashboard UI, and we haven't got any issue in the parent Child Handshake using dbutils.notebook.exit(json) command
however, with the same code, we could see that the streaming window popped for 05.25AM run and this time child notebook, doesn't send the dbutils.notebook.exit(json) to the parent.
