cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How to check my streaming job's metrics?

jose_gonzalez
Moderator
Moderator

I would like to know if there is a way to keep track of my running streaming job.

3 REPLIES 3

User16783854357
New Contributor III

Additionally, if you want to monitor this programmatically , you can attach a streaming listener to your query and review the query progress event . This link provides more details.

User16869510359
Esteemed Contributor

Streaming metrics are available/exposed mainly through 3 ways:

  1. Streaming UI, which is available from Spark 3/DBR 7
  2. Streaming listener/Observable metrics API
  3. Spark driver logs. Search for the string "Streaming query made progress". The metrics are logged in the Spark driver logs as soon as the batch is completed.
Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.