topic How to set size of Parquet output files ? in Data Engineering

How to set size of Parquet output files ?

richard1_558848 — Tue, 19 May 2015 09:57:02 GMT

I'm using Parquet for format to store Raw Data. Actually the part file are stored on S3

I would like to control the file size of each parquet part file.

I try this

sqlContext.setConf("spark.parquet.block.size", SIZE.toString)

sqlContext.setConf("spark.dfs.blocksize", SIZE.toString)

But not seems to work. Could you help me?

Re: How to set size of Parquet output files ?

richard1_558848 — Wed, 20 May 2015 18:36:32 GMT

Any Information ?

Re: How to set size of Parquet output files ?

__rake — Tue, 30 Jun 2015 14:02:20 GMT

Try this (in 1.4.0):

val blockSize = 1024 * 1024 * 16      // 16MB
sc.hadoopConfiguration.setInt( "dfs.blocksize", blockSize )
sc.hadoopConfiguration.setInt( "parquet.block.size", blockSize )

Where sc is your SparkContext (not SQLContext).

Not that there also appears to be "page size" and "dictionary page size" parameters that interact with the block size; e.g., page size should not exceed block size. I have them all with the exact same value, and that got me through.

It looks like Spark will allocate 1 block in memory for every Parquet partition you output, so if you are creating a large number of Parquet partitions you could quickly hit OutOfMemory errors.

Re: How to set size of Parquet output files ?

manjeet_chandho — Thu, 05 Jan 2017 05:58:14 GMT

Hi All can anyone tell me what is the default Raw Group size while writing via SparkSql