Hello all!

I'm running a simple read noop query where I read a specific partition of a delta table that looks like this:

With the default configuration, I read the data in 12 partitions, which makes sense as the files that are more than 128MB are split.

When I configure "spark.sql.files.maxPartitionBytes" (or "spark.files.maxPartitionBytes") to 64MB, I do read with 20 partitions as expected. THOUGH the extra partitions are empty (or some kilobytes)

I have tested with "spark.sql.adaptive.enabled" set to true and false without any change in the behaviour.

Any thoughts why this is happening and how force spark to read in smaller partitions?

Thank you in advance for your help!