This all depends on the dataset size and specific use cases, but, in general, we've seen that Parquet partitions of about 1GB are optimal. For smaller datasets, however, this large partition size may limit parallelism as tasks operate on individual partitions in parallel, so please keep that in mind.

Hi, I'm using the orc file format instead of Parquet, would you recommend a similar size for that too, or perhaps something larger?

How do you set this size ?