Incorrect dropped rows count in DLT Event log
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-14-2023 05:23 AM
Hello,
I'm using a DLT pipeline with expectations
expect_or_drop(...)
To test it, I added files that contain records that should be dropped, and indeed when running the pipeline I can see some rows were dropped.
However when looking at the DLT Event logs, in the field
details.flow_progress.data_quality.dropped_recordsI always get 0, and the number of failing rows are only available in the field
details.flow_progress.data_quality.expectations.failed_records
Is this the expected behavior ?