Walter_C
Databricks Employee
Databricks Employee

The commitBatch metric is a part of the overall triggerExecution time, which encompasses all stages of planning and executing the microbatch, including committing the batch data and updating offsets.

The commitBatch metric may not always be present in every example. Its presence depends on the specific implementation and the metrics that are being tracked for that particular streaming query. For instance, in the examples you mentioned:

  • The rate source to Delta Lake example does not include commitBatch because it may not be relevant or tracked for that specific query.
  • The Kafka-to-Kafka example also does not include commitBatch, possibly due to differences in how metrics are collected or reported for Kafka sinks.