does stream.stop() generates "ERROR: Query termination received for []" automatically?

xhudik — Fri, 27 Sep 2024 13:31:15 GMT

Whenever code contains stream.stop() in STDERR (in cluster logs) I get an error like:

ERROR: Query termination received for [id=b7e14d07-f8ad-4ae6-99de-8a7cbba89d86, runId=5c01fd71-2d93-48ca-a53c-5f46fab726ff]

No other message, even if I try to try-catch and get some error stack no success (always only this 1 line error). When I take a look on Log4j output, there is an info instead of error:

24/09/27 12:42:46 INFO DatabricksStreamingQueryListener: Query termination received for [id=b7e14d07-f8ad-4ae6-99de-8a7cbba89d86, runId=5c01fd71-2d93-48ca-a53c-5f46fab726ff]

I'm wondering:

does stream.stop() generates "ERROR: Query termination received for []" automatically?
Is ERROR: Query termination received for [] dangerous, or it is just ans info stream was closed?

Re: does stream.stop() generates "ERROR: Query termination received for []" automatically?

MuthuLakshmi — Thu, 07 Nov 2024 16:38:59 GMT

@xhudik
does stream.stop() generates "ERROR: Query termination received for []" automatically?
Yes, this is generated when there is stream.stop() in stdderIs

ERROR: Query termination received for [] dangerous, or it is just ans info stream was closed?
The "Query termination received" message that is not preceded by an Exception is not a concern, since this message is simply triggered by streamingContext.stop(), which stops the execution of streams after processing all incoming data. This is considered a graceful stop scenario and a behavior by design for logs.

topic does stream.stop() generates "ERROR: Query termination received for []" automatically? in Data Engineering

does stream.stop() generates "ERROR: Query termination received for []" automatically?

Re: does stream.stop() generates "ERROR: Query termination received for []" automatically?