topic Re: What is an optimal size for file partitions using Parquet? in Data Engineering https://community.databricks.com/t5/data-engineering/what-is-an-optimal-size-for-file-partitions-using-parquet/m-p/30882#M22440 <P>This all depends on the dataset size and specific use cases, but, in general, we've seen that Parquet partitions of about 1GB are optimal. For smaller datasets, however, this large partition size may limit parallelism as tasks operate on individual partitions in parallel, so please keep that in mind. </P> Wed, 18 Feb 2015 21:26:44 GMT __Databricks_Su 2015-02-18T21:26:44Z What is an optimal size for file partitions using Parquet? https://community.databricks.com/t5/data-engineering/what-is-an-optimal-size-for-file-partitions-using-parquet/m-p/30881#M22439 Wed, 18 Feb 2015 21:26:44 GMT https://community.databricks.com/t5/data-engineering/what-is-an-optimal-size-for-file-partitions-using-parquet/m-p/30881#M22439 __Databricks_Su 2015-02-18T21:26:44Z Re: What is an optimal size for file partitions using Parquet? https://community.databricks.com/t5/data-engineering/what-is-an-optimal-size-for-file-partitions-using-parquet/m-p/30882#M22440 <P>This all depends on the dataset size and specific use cases, but, in general, we've seen that Parquet partitions of about 1GB are optimal. For smaller datasets, however, this large partition size may limit parallelism as tasks operate on individual partitions in parallel, so please keep that in mind. </P> Wed, 18 Feb 2015 21:26:44 GMT https://community.databricks.com/t5/data-engineering/what-is-an-optimal-size-for-file-partitions-using-parquet/m-p/30882#M22440 __Databricks_Su 2015-02-18T21:26:44Z Re: What is an optimal size for file partitions using Parquet? https://community.databricks.com/t5/data-engineering/what-is-an-optimal-size-for-file-partitions-using-parquet/m-p/30883#M22441 <P></P> <P>Hi, I'm using the orc file format instead of Parquet, would you recommend a similar size for that too, or perhaps something larger? </P> <P></P> Mon, 18 May 2015 18:07:34 GMT https://community.databricks.com/t5/data-engineering/what-is-an-optimal-size-for-file-partitions-using-parquet/m-p/30883#M22441 msj50 2015-05-18T18:07:34Z Re: What is an optimal size for file partitions using Parquet? https://community.databricks.com/t5/data-engineering/what-is-an-optimal-size-for-file-partitions-using-parquet/m-p/30884#M22442 <P></P> <P>How do you set this size ?</P> <P></P> Wed, 20 May 2015 11:59:49 GMT https://community.databricks.com/t5/data-engineering/what-is-an-optimal-size-for-file-partitions-using-parquet/m-p/30884#M22442 richard1_558848 2015-05-20T11:59:49Z