I have S3 as a data source containing sample TPC dataset (10G, 100G).

I want to convert that into parquet files with an average size of about ~256MiB. What configuration parameter can I use to set that?

I also need the data to be partitioned. And within each partition column, the files should be split based on the average size.

If I set the coalesce option `df.coalesce(1)`, then it only creates 1 file in each partition column.

I've also tried setting the parameters (based off Google search)

```

df.write.option("maxRecordsPerFile", 6000000)

df.write.option("parquet.block.size", 256 * 1024 * 1024)

```

But that didn't help either. Any suggestions?