cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Shiva_Dsouz
by New Contributor II
  • 934 Views
  • 1 replies
  • 1 kudos

How to get spark streaming metrics like input rows, processed rows and batch duration to Prometheus for monitoring

I have been reading this article https://www.databricks.com/session_na20/native-support-of-prometheus-monitoring-in-apache-spark-3-0 and it has been mentioned that we can get the spark streaming metrics like input rows, processing rate and batch dura...

  • 934 Views
  • 1 replies
  • 1 kudos
Latest Reply
SS2
Valued Contributor
  • 1 kudos

I think you can use spark UI to see deep level details ​

  • 1 kudos
Erik
by Valued Contributor II
  • 1845 Views
  • 6 replies
  • 8 kudos

Expected latency / batch duration for a simple streaming job?

What are "reasonable"/"normal" batch durations for easy (no real processing, just adding a few simple fields) streaming jobs into/from delta lake? We have set up a simple test case here where we are streaming from azure event hub generating a new mes...

  • 1845 Views
  • 6 replies
  • 8 kudos
Latest Reply
Kaniz
Community Manager
  • 8 kudos

Hi @Erik Parmann​  , Just a friendly follow-up. Do you still need help, or does my response help you to find the solution? Please let us know.

  • 8 kudos
5 More Replies
Labels