@ranged_coop Regarding your questions:

  • Is there any setting that needs to be enable to fix this?
    There is no setting to change this behavior, as it is a design decision and not an issue. Looks like you're referring to checkpointing. These are the docs: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-fai...
  • Will it cause any issues with the history?
    Spark structured streaming provides exactly-once processing guarantees. How you process the data depends on the logic implemented in your state management.

Structured Streaming Programming Guide: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#structured-streamin...

Best regards,

Raphael Balogo
Sr. Technical Solutions Engineer
Databricks