- 3280 Views
- 7 replies
- 8 kudos
Expected latency / batch duration for a simple streaming job?
What are "reasonable"/"normal" batch durations for easy (no real processing, just adding a few simple fields) streaming jobs into/from delta lake? We have set up a simple test case here where we are streaming from azure event hub generating a new mes...
- 3280 Views
- 7 replies
- 8 kudos
- 8 kudos
Hi @Erik Parmann​ , Just a friendly follow-up. Do you still need help, or does my response help you to find the solution? Please let us know.
- 8 kudos
- 1556 Views
- 1 replies
- 1 kudos
How to get spark streaming metrics like input rows, processed rows and batch duration to Prometheus for monitoring
I have been reading this article https://www.databricks.com/session_na20/native-support-of-prometheus-monitoring-in-apache-spark-3-0 and it has been mentioned that we can get the spark streaming metrics like input rows, processing rate and batch dura...
- 1556 Views
- 1 replies
- 1 kudos
- 1 kudos
I think you can use spark UI to see deep level details ​
- 1 kudos