Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-09-2024 08:36 AM - edited 09-09-2024 08:44 AM
@ranged_coop Regarding your questions:
- Is there any setting that needs to be enable to fix this?
There is no setting to change this behavior, as it is a design decision and not an issue. Looks like you're referring to checkpointing. These are the docs: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#recovering-from-fai... - Will it cause any issues with the history?
Spark structured streaming provides exactly-once processing guarantees. How you process the data depends on the logic implemented in your state management.
Structured Streaming Programming Guide: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#structured-streamin...
Best regards,
Raphael Balogo
Sr. Technical Solutions Engineer
Databricks
Raphael Balogo
Sr. Technical Solutions Engineer
Databricks