How to check my streaming job's metrics?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-04-2021 11:41 AM
I would like to know if there is a way to keep track of my running streaming job.
Labels:
- Labels:
-
Metrics
-
Spark-streaming
3 REPLIES 3
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2021 11:08 AM
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2021 04:05 PM
Additionally, if you want to monitor this programmatically , you can attach a streaming listener to your query and review the query progress event . This link provides more details.
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-23-2021 06:20 AM
Streaming metrics are available/exposed mainly through 3 ways:
- Streaming UI, which is available from Spark 3/DBR 7
- Streaming listener/Observable metrics API
- Spark driver logs. Search for the string "Streaming query made progress". The metrics are logged in the Spark driver logs as soon as the batch is completed.

