Additionally, if you want to monitor this programmatically , you can attach a streaming listener to your query and review the query progress event . This link provides more details.
Streaming metrics are available/exposed mainly through 3 ways:
Streaming UI, which is available from Spark 3/DBR 7
Streaming listener/Observable metrics API
Spark driver logs. Search for the string "Streaming query made progress". The metrics are logged in the Spark driver logs as soon as the batch is completed.
