cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to get spark streaming metrics like input rows, processed rows and batch duration to Prometheus for monitoring

Shiva_Dsouz
New Contributor II

I have been reading this article

https://www.databricks.com/session_na20/native-support-of-prometheus-monitoring-in-apache-spark-3-0 and it has been mentioned that we can get the spark streaming metrics like input rows, processing rate and batch duration into prometheus.

I was able to get host/infra metrics like memory, disk etc via below API.

https://eastus-c3.databricks.net/driver-proxy-api/o/<org-id>/<cluster-id>/40001/metrics/executors/pr...

I couldnt find any apis or references to get the streaming metrics, processing info etc. Any help on how to get those streaming UI metrics to prometheus ?

Spark configs being set on cluster:

spark.ui.prometheus.enabled true

spark.sql.streaming.metricsEnabled true

1 REPLY 1

SS2
Valued Contributor

I think you can use spark UI to see deep level details โ€‹

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.