- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
yesterday - last edited yesterday
Thanks for the thorough reply.
>> The checkpoint only has metadata, commits and offsets, with no state folder.
It is a bit counterintuitive to have a bunch of stateful information stored as metadata, commits, and offsets, yet to NOT be able to use a "statestore api" to inspect that information.
>> For a durable audit trail, write batch_id and row counts to a table from foreachBatch, or persist progress events with a StreamingQueryListener.
Thanks for the reference to "StreamingQueryListener". That is the sort of turnkey API that I was missing. It wasn't coming up in my searches.
I'm guessing designers of spark streaming did not intend for users to tear apart the checkpointLocation on our own (although it still seems important to have that ability in a pinch). I was surprised NOT to find a github library or example that creates such audit trails at this level of detail. Maybe with your additional keywords (batch_id, reservoirVersion) then I will find something....
FYI, I posted a similar question elsewhere. My underlying goals are not only to be able to audit the checkpointLocation but also to have a strategy for recovering my streams in the event that my checkpointLocation became corrupted one day (for reasons outside my control, for example if I store it on a less dependable location than the place where the deltalake tables live.). I learned about the startingVersion/startingTimestamp that can be used for recovery. So if I am able to do some checkpointLocation auditing, and make sense of the results, then it will give me better shot at setting startingVersion/startingTimestamp for a point-in-time recovery.