https://community.databricks.com/t5/data-engineering/how-to-check-my-streaming-job-s-metrics/m-p/26381#M18451 <P>Streaming metrics are available/exposed mainly through 3 ways:</P><OL><LI>Streaming UI, which is available from Spark 3/DBR 7</LI><LI>Streaming listener/Observable metrics API</LI><LI>Spark driver logs. Search for the string "Streaming query made progress". The metrics are logged in the Spark driver logs as soon as the batch is completed. </LI></OL> Wed, 23 Jun 2021 13:20:01 GMT brickster_2018 2021-06-23T13:20:01Z https://community.databricks.com/t5/data-engineering/how-to-check-my-streaming-job-s-metrics/m-p/26378#M18448 <P>I would like to know if there is a way to keep track of my running streaming job. </P> Fri, 04 Jun 2021 18:41:56 GMT https://community.databricks.com/t5/data-engineering/how-to-check-my-streaming-job-s-metrics/m-p/26378#M18448 jose_gonzalez 2021-06-04T18:41:56Z https://community.databricks.com/t5/data-engineering/how-to-check-my-streaming-job-s-metrics/m-p/26379#M18449 <P><A href="https://databricks.com/blog/2020/07/29/a-look-at-the-new-structured-streaming-ui-in-apache-spark-3-0.html" target="test_blank">https://databricks.com/blog/2020/07/29/a-look-at-the-new-structured-streaming-ui-in-apache-spark-3-0.html</A></P> Mon, 07 Jun 2021 18:08:06 GMT https://community.databricks.com/t5/data-engineering/how-to-check-my-streaming-job-s-metrics/m-p/26379#M18449 User16783855534 2021-06-07T18:08:06Z https://community.databricks.com/t5/data-engineering/how-to-check-my-streaming-job-s-metrics/m-p/26380#M18450 <P>Additionally, if you want to monitor this programmatically , you can attach a streaming listener to your query and review the query progress event . <A href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#monitoring-streaming-queries" alt="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#monitoring-streaming-queries" target="_blank">This link</A> provides more details. </P> Mon, 07 Jun 2021 23:05:51 GMT https://community.databricks.com/t5/data-engineering/how-to-check-my-streaming-job-s-metrics/m-p/26380#M18450 User16783854357 2021-06-07T23:05:51Z https://community.databricks.com/t5/data-engineering/how-to-check-my-streaming-job-s-metrics/m-p/26381#M18451 <P>Streaming metrics are available/exposed mainly through 3 ways:</P><OL><LI>Streaming UI, which is available from Spark 3/DBR 7</LI><LI>Streaming listener/Observable metrics API</LI><LI>Spark driver logs. Search for the string "Streaming query made progress". The metrics are logged in the Spark driver logs as soon as the batch is completed. </LI></OL> Wed, 23 Jun 2021 13:20:01 GMT https://community.databricks.com/t5/data-engineering/how-to-check-my-streaming-job-s-metrics/m-p/26381#M18451 brickster_2018 2021-06-23T13:20:01Z