cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Shiva_Dsouz
by New Contributor II
  • 1081 Views
  • 1 replies
  • 1 kudos

How to get spark streaming metrics like input rows, processed rows and batch duration to Prometheus for monitoring

I have been reading this article https://www.databricks.com/session_na20/native-support-of-prometheus-monitoring-in-apache-spark-3-0 and it has been mentioned that we can get the spark streaming metrics like input rows, processing rate and batch dura...

  • 1081 Views
  • 1 replies
  • 1 kudos
Latest Reply
SS2
Valued Contributor
  • 1 kudos

I think you can use spark UI to see deep level details ​

  • 1 kudos
TyronZerafa
by New Contributor II
  • 1247 Views
  • 0 replies
  • 2 kudos

Integrating with Prometheus

How can I integrate Databricks clusters with Prometheus? I tried adding the following Spark property to my cluster but cannot find the Prometheus metrics endpoints. Any thoughts? spark.ui.prometheus.enabled = true

  • 1247 Views
  • 0 replies
  • 2 kudos
Labels