In my team, we decided to move from spark streaming to structured streaming, mainly cause it says that it's legacy and we want to benefit new features from structured streaming.
However we have an issue with committing offsets.
Previously on spark streaming, we were creating DStream, doing some transformation/action and commiting offsets at the end which was totally fine.
On Structured Streaming, we dont want to rely on spark checkpointing (issues with application upgrade and for example the consumer group used can't be monitored to see if there is lag (cause there's no offset commited on kafka side).
For now, we are using an external tool (Redis) to store those offsets and start from them in case of application restart but it's not convenient.
Is there a way to use native kafka consumer group commit with structured streaming and restart from that ?
Thanks for your hints!
Erwan
