-werners-
Esteemed Contributor III

The AQE will only kick in when you are actually doin transformations (shuffle/broadcast) and it will try to optimize the partition size:

https://docs.microsoft.com/en-us/azure/databricks/spark/latest/spark-sql/aqe#dynamically-coalesce-pa...

AFAIK the read-partitionsize is indeed defined by maxPartitionBytes.

Now, I do recall a topic on stackoverflow where someone asks a similar question.

And there they mention the compression coded also matters.

Chances are you use snappy compression. If that is the case, the partition size might be defined by the row group size of the parquet files.

https://stackoverflow.com/questions/32382352/is-snappy-splittable-or-not-splittable

http://boristyukin.com/is-snappy-compressed-parquet-file-splittable/

Also David Vrba mentions the compression used too:

https://stackoverflow.com/questions/62648621/spark-sql-files-maxpartitionbytes-not-limiting-max-size...

View solution in original post