Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-12-2026 03:04 AM
Could someone from Databricks answer on that question.
On top of the above numInputRows metric is not (imho) reliable information as well.
Checked several stateless streams (simple foreachBatch function with extraction and straightforward validation where "bad" records are moved to "quarantine" table), compared endOffsets minus startOffsets with numInputRows metric, as well as, with actual data processed to the target table:
- endOffsets minus startOffsets is the same as the actual number of records committed to target table - if we will not take into account records that were rolled back by consumer - i.e. some offsets are not processed to target table.
- numInputRows in 95% of cases is 3 or 5 times bigger than the actual number of processed records (i.e. records sunk to target table - does that mean that batch was re-started 3 or 5 times? it would explain a lot, but there some rare scenarious
- where numInputRows does not seems to be in exact correlation with actual processed number of records. For instance, event with numInputRows = 10523, actual committed records for that particular batch in the target table = 3198 and number of processed offsets is 3224 (looks like there were uncommitted events on producer)
Hard to find logic here...