Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-05-2021 11:45 AM
Situation: we have one partion per date, and it just so happens that each partition ends up (after optimize) as *a single* 128mb file. We partition on date, and zorder on userid, and our query is something like "find max value of column A where userid=X and date>=somedate".
Does zordering help in any way in this scenario? It is clear that we will have to read every partition after $somedate, but does the zordering on userid somehow help spark when reading inside each of those partitions (remember that each partition is a single file), or do we have to read *and scan* all 128mb of each of the remaining partitions even when we zoptimize?
Labels:
- Labels:
-
Reading
-
Z-ordering