Databricks Community

f2008700 · 06-04-2023

I have S3 as a data source containing sample TPC dataset (10G, 100G).I want to convert that into parquet files with an average size of about ~256MiB. What configuration parameter can I use to set that?I also need the data to be partitioned. And withi...

f2008700 · 06-13-2023

Thanks!Will give it a try.

f2008700 · 06-07-2023

Thanks for pointing it out. My ask if actually the opposite. I want to ensure that the basic size of parquet file is not too small, but rather more than a certain basic size/ or contains a minimum number of records.

Databricks Community

User Stats

User Activity

Configuring average parquet file size

Re: Configuring average parquet file size

Re: Configuring average parquet file size