cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Shiva_Dsouz
by • New Contributor II
  • 2040 Views
  • 1 replies
  • 1 kudos

How to get spark streaming metrics like input rows, processed rows and batch duration to Prometheus for monitoring

I have been reading this article https://www.databricks.com/session_na20/native-support-of-prometheus-monitoring-in-apache-spark-3-0 and it has been mentioned that we can get the spark streaming metrics like input rows, processing rate and batch dura...

Data Engineering
Reply
  • 2040 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
SS2
Valued Contributor
  • 1 kudos

I think you can use spark UI to see deep level details ​

  • 1 kudos
jose_gonzalez
by • Databricks Employee
  • 2702 Views
  • 3 replies
  • 0 kudos

How to check my streaming job's metrics?

I would like to know if there is a way to keep track of my running streaming job.

Data Engineering
Reply
  • 2702 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
brickster_2018
Databricks Employee
  • 0 kudos

Streaming metrics are available/exposed mainly through 3 ways:Streaming UI, which is available from Spark 3/DBR 7Streaming listener/Observable metrics APISpark driver logs. Search for the string "Streaming query made progress". The metrics are logged...

  • 0 kudos
2 More Replies
User16789201666
by • Databricks Employee
  • 1402 Views
  • 1 replies
  • 0 kudos

For our streaming jobs, I am planning to purge the data older than 60 days from our raw layer where we dump the data into delta table from kinesis stream in json format. Do you see any problem in parallelly doing the purging on older data while streaming

Data Engineering
Reply
  • 1402 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16789201666
Databricks Employee
  • 0 kudos

There isn’t a problem purging old data. When using auto loader it’ll take into account new data being added.

  • 0 kudos
Labels
Top