cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Shiva_Dsouz
by • New Contributor II
  • 1799 Views
  • 1 replies
  • 1 kudos

How to get spark streaming metrics like input rows, processed rows and batch duration to Prometheus for monitoring

I have been reading this article https://www.databricks.com/session_na20/native-support-of-prometheus-monitoring-in-apache-spark-3-0 and it has been mentioned that we can get the spark streaming metrics like input rows, processing rate and batch dura...

Data Engineering
Reply
  • 1799 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
SS2
Valued Contributor
  • 1 kudos

I think you can use spark UI to see deep level details ​

  • 1 kudos
TyronZerafa
by • New Contributor II
  • 1670 Views
  • 0 replies
  • 2 kudos

Integrating with Prometheus

How can I integrate Databricks clusters with Prometheus? I tried adding the following Spark property to my cluster but cannot find the Prometheus metrics endpoints. Any thoughts? spark.ui.prometheus.enabled = true

Data Engineering
Reply
  • 1670 Views
  • 0 replies
  • 2 kudos
Labels