How to get spark streaming metrics like input rows, processed rows and batch duration to Prometheus for monitoring

[Shiva_Dsouz]

I have been reading this article

https://www.databricks.com/session_na20/native-support-of-prometheus-monitoring-in-apache-spark-3-0 and it has been mentioned that we can get the spark streaming metrics like input rows, processing rate and batch duration into prometheus.

I was able to get host/infra metrics like memory, disk etc via below API.

https://eastus-c3.databricks.net/driver-proxy-api/o/<org-id>/<cluster-id>/40001/metrics/executors/pr...

I couldnt find any apis or references to get the streaming metrics, processing info etc. Any help on how to get those streaming UI metrics to prometheus ?

Spark configs being set on cluster:

spark.ui.prometheus.enabled true

spark.sql.streaming.metricsEnabled true

[SS2]

I think you can use spark UI to see deep level details ​