cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Shiva_Dsouz
by • New Contributor II
  • 968 Views
  • 1 replies
  • 1 kudos

How to get spark streaming metrics like input rows, processed rows and batch duration to Prometheus for monitoring

I have been reading this article https://www.databricks.com/session_na20/native-support-of-prometheus-monitoring-in-apache-spark-3-0 and it has been mentioned that we can get the spark streaming metrics like input rows, processing rate and batch dura...

Data Engineering
Reply
  • 968 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
SS2
Valued Contributor
  • 1 kudos

I think you can use spark UI to see deep level details ​

  • 1 kudos
jose_gonzalez
by • Moderator
  • 1219 Views
  • 3 replies
  • 0 kudos

How to check my streaming job's metrics?

I would like to know if there is a way to keep track of my running streaming job.

Data Engineering
Reply
  • 1219 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
User16869510359
Esteemed Contributor
  • 0 kudos

Streaming metrics are available/exposed mainly through 3 ways:Streaming UI, which is available from Spark 3/DBR 7Streaming listener/Observable metrics APISpark driver logs. Search for the string "Streaming query made progress". The metrics are logged...

  • 0 kudos
2 More Replies
User16789201666
by • Contributor II
  • 613 Views
  • 1 replies
  • 0 kudos

For our streaming jobs, I am planning to purge the data older than 60 days from our raw layer where we dump the data into delta table from kinesis stream in json format. Do you see any problem in parallelly doing the purging on older data while streaming

Data Engineering
Reply
  • 613 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
User16789201666
Contributor II
  • 0 kudos

There isn’t a problem purging old data. When using auto loader it’ll take into account new data being added.

  • 0 kudos
Labels