Monitoring a Streaming Job

NandiniN — Sat, 01 Jun 2024 21:46:31 GMT

If you have a streaming job, you need to check the batch metrics to be able to understand the stream progress.

However, here are some other suggestions which we can use to monitor a streaming job and be stuck in a "hung" state.

Streaming Listeners
spark.databricks.hangingTaskDetector.assumeTaskMakingProgressForZeroMetrics to false (Default=true).
If it is a stateless stream, we can also turn on spark.speculation by setting it to true.
Set job notifications.

And of course don't forget to collect a thread dump, before you cancel the stream and raise a support ticket to investigate further.

topic Monitoring a Streaming Job in Community Articles