topic How to monitor Kafka consumption / lag when working with spark structured streaming? in Data Engineering

How to monitor Kafka consumption / lag when working with spark structured streaming?

Maatari — Thu, 11 Jul 2024 17:32:21 GMT

I have just find out spark structured streaming do not commit offset to kafka but use its internal checkpoint system and that there is no way to visualize its consumption lag in typical kafka UI

- https://community.databricks.com/t5/data-engineering/can-we-commit-offset-in-spark-structured-streaming-in-databricks/td-p/71900#:~:text=Spark%20Structured%20Streaming%20manages%20which,a%20spark%20structured%20streaming%20query ?

Lag being important in stream processing, I can't imagine that the community did not come up with workaround to help with consumer lag tracking, but so far i could not find any out of the box solutions.

In any case, as i do not want to reinvent the will, I won't if anyone can share solution either out of the box or custom that people typically uses for this ?

Re: How to monitor Kafka consumption / lag when working with spark structured streaming?

RishabhTiwari07 — Fri, 19 Jul 2024 12:20:21 GMT

Hi @Maatari ,

Thank you for reaching out to our community! We're here to help you.

To ensure we provide you with the best support, could you please take a moment to review the response and choose the one that best answers your question? Your feedback not only helps us assist you better but also benefits other community members who may have similar questions in the future.

If you found the answer helpful, consider giving it a kudo. If the response fully addresses your question, please mark it as the accepted solution. This will help us close the thread and ensure your question is resolved.

We appreciate your participation and are here to assist you further if you need it!

Thanks,

Rishabh