Re: Graceful shutdown - stopping stream at the end...

itt · ‎02-10-2025

Hey, thanks for the replay

It still seems to cut at the middle of the stream.

For a simple test, i created the following:

onend = -1

on_entry = -1

def foreachbatch(df, df_id😞

global onend

global on_entry

on_entry = df_id

time.sleep(5)

df.count()

dff = spark.table("some_table").limit(1)

dff.write.insertInto("some_table", overwrite=False)

onend = df_id

spark.conf.set("spark.streaming.stopGracefullyOnShutdown", "true")

stream_reader = (

spark.readStream.format("delta").option("ignoreChanges", "true").option("spark.streaming.stopGracefullyOnShutdown", "true")

)

stream_reader = stream_reader.table("some_table")

writer = stream_reader.writeStream.trigger(

processingTime="1 seconds"

).option("maxBytesPerTrigger", 1).queryName("thestream")

writer = writer.outputMode("append").format("delta").foreachBatch(foreachbatch)

query = writer.start()

and to stop i do:

query.stop()

And the values are not changing for "onend"

of curse this is just for testing, but still it seems like it does not work