Her you can find some info. Basically it boils down to: reading a file = overhead. You want to minimize overhead without stressing the workers too much (gigantic partitions).
In this article, I shall tell you different ways to solve the large number of small files problem. This article will help Data Engineers to optimize the output storage of their Spark applications.
