Graceful shutdown - stopping stream at the end of microbatch

[itt]

Im trying to create a system where i let spark finish the current microbatch, and letting it know it should stop after it.

The reason is that i don't want to re-calcualte a microbatch with "forcefully" stopping a stream.

Is there a way spark/databricks already implemented this?

My current approach is just raising exception at the start of the microbatch, if an indication of gracefull stop was given.

Thanks

[itt]

Hey, thanks for the replay

It still seems to cut at the middle of the stream.

For a simple test, i created the following:

onend = -1

on_entry = -1

def foreachbatch(df, df_id😞

    global onend

    global on_entry

    on_entry = df_id

    time.sleep(5)

    df.count()

    dff = spark.table("some_table").limit(1)

    dff.write.insertInto("some_table", overwrite=False)

    onend = df_id

spark.conf.set("spark.streaming.stopGracefullyOnShutdown", "true")

stream_reader = (

    spark.readStream.format("delta").option("ignoreChanges", "true").option("spark.streaming.stopGracefullyOnShutdown", "true")

)

stream_reader = stream_reader.table("some_table")

writer = stream_reader.writeStream.trigger(

    processingTime="1 seconds"

).option("maxBytesPerTrigger", 1).queryName("thestream")

writer = writer.outputMode("append").format("delta").foreachBatch(foreachbatch)

query = writer.start()

and to stop i do:

query.stop()

And the values are not changing for "onend"

of curse this is just for testing, but still it seems like it does not work