How to get spark streaming metrics like input rows, processed rows and batch duration to Prometheus for monitoring

Shiva_Dsouz — Thu, 24 Nov 2022 14:22:13 GMT

I have been reading this article

https://www.databricks.com/session_na20/native-support-of-prometheus-monitoring-in-apache-spark-3-0 and it has been mentioned that we can get the spark streaming metrics like input rows, processing rate and batch duration into prometheus.

I was able to get host/infra metrics like memory, disk etc via below API.

https://eastus-c3.databricks.net/driver-proxy-api/o/<org-id>/<cluster-id>/40001/metrics/executors/prometheus

I couldnt find any apis or references to get the streaming metrics, processing info etc. Any help on how to get those streaming UI metrics to prometheus ?

Spark configs being set on cluster:

spark.ui.prometheus.enabled true

spark.sql.streaming.metricsEnabled true

Re: How to get spark streaming metrics like input rows, processed rows and batch duration to Prometheus for monitoring

SS2 — Tue, 29 Nov 2022 19:37:37 GMT

I think you can use spark UI to see deep level details

topic How to get spark streaming metrics like input rows, processed rows and batch duration to Prometheus for monitoring in Data Engineering

How to get spark streaming metrics like input rows, processed rows and batch duration to Prometheus for monitoring

Re: How to get spark streaming metrics like input rows, processed rows and batch duration to Prometheus for monitoring