- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-23-2021 02:36 AM
The AQE will only kick in when you are actually doin transformations (shuffle/broadcast) and it will try to optimize the partition size:
AFAIK the read-partitionsize is indeed defined by maxPartitionBytes.
Now, I do recall a topic on stackoverflow where someone asks a similar question.
And there they mention the compression coded also matters.
Chances are you use snappy compression. If that is the case, the partition size might be defined by the row group size of the parquet files.
https://stackoverflow.com/questions/32382352/is-snappy-splittable-or-not-splittable
http://boristyukin.com/is-snappy-compressed-parquet-file-splittable/
Also David Vrba mentions the compression used too: