Hi @Maatari ,

In spark structured streaming, current offset information is written to checkpoint files continuously. You can create piece of code that will extract information from checkpoint files aobut currently consumed offset, extract offset from Kafka and compare it.

As an example, look at below article. Unfortunately, I don't know anything about out of the box solution for this kind of problem.

Monitoring Spark Structured Streaming/Kafka Offsets with Prometheus and Grafana | by Lim Yow Cheng |...

PS. There is Kafka offset commiter for Spark Structured Streaming, but last commits are from 4 years ago 🙂

HeartSaVioR/spark-sql-kafka-offset-committer: Kafka offset committer for structured streaming query ...