Hey, thanks for the replay

It still seems to cut at the middle of the stream.

For a simple test, i created the following:

Checking.

There is no built-in Spark or Databricks method to gracefully stop a Structured Streaming query specifically at the end of the current microbatch, but several community and expert discussions propose common strategies to achieve this:

Official and Common Approaches

• The standard StreamingQuery.stop() method stops the execution thread and cancels jobs, but it does not guarantee waiting for the current microbatch to finish—all jobs are cancelled at once, so in-flight microbatch processing may be interrupted and could result in partial data or the need for recalculation upon restart.​

• For traditional Spark Streaming (DStreams, not Structured Streaming), there is a graceful stop option (StreamingContext.stop(stopSparkContext, stopGracefully)), which waits for processing of all received data. However, this is not available for Structured Streaming.​

Community Solutions

• A common workaround is to signal for a graceful shutdown (e.g., by creating a marker file, listening on a socket, etc.) and then wait for the current trigger (microbatch) to complete before calling stop() on the StreamingQuery object. This should be done outside the microbatch processing code to avoid deadlocks, as stopping the query inside the microbatch can cause it to never finish.​

• Another method is to periodically check if the stream's status (StreamingQuery.status.isTriggerActive) is false—which would indicate that the microbatch has completed—and only then call .stop() when an external termination signal is received.​

Example Approach

• Use an external mechanism (file, API, REST endpoint) to signal when a graceful stop is required.

• Monitor the status of the streaming query in a separate thread or loop.

• After receiving the stop signal, wait until the current microbatch has finished (isTriggerActive == false), then call .stop().​

Why Not Raise Exceptions

• Raising exceptions inside the microbatch task can interrupt it mid-processing, risking partial computation or data duplication/loss, which is why most best practices recommend monitoring and stopping externally.​

Summary Table: Structured Streaming Stop Methods

Consult the Spark or Databricks documentation for updates, as enhancements may eventually address this scenario natively. For immediate best results, use external coordination and status monitoring as described above.​