numSourceRows greater than expected
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-08-2023 02:51 AM
Hey
I am doing an upsert of a source DataFrame into a target table. Before said upsert, I print out the source DataFrame's row count, which is a bit smaller than what `numSourceRows` says after the operation completes and I check the operationMetrics. Two things occurred to me as to why this is happening:
- The matching condition is being matched more than once (it's not, I checked. And it wouldn't make sense that this affects the field, according to what the Doc says)
- Some rows are being modified because they are written in the same page as the other truly modified rows (still doesn't make sense).
What situations might cause this?
- Labels:
-
Target Table
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-08-2023 04:18 AM
That is indeed interesting, never looked into it.
I just searched on the delta github space and found some commits that show there is a bit more to it than just a count:
https://github.com/delta-io/delta/commit/d2804cb92a7e36863144c7be9c55df1c6f1c1a1e
https://github.com/delta-io/delta/commit/8624b92ddd8d47f98e91b88b19b6d4af2e09033b
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-01-2023 10:54 AM
could you share your code snippet please? also share the expected output.