topic Re: Writing part files in single text file in Data Engineering

Writing part files in single text file

Manthansingh — Thu, 01 Aug 2024 13:28:42 GMT

i want to write all my part file into a single text file is there anything i can do

Re: Writing part files in single text file

Witold — Thu, 01 Aug 2024 13:49:27 GMT

coalesce with one partition might be your friend:

( df .coalesce(1) .write.format('csv') .option('header', 'true') .save('one-file.csv') )

Re: Writing part files in single text file

Edthehead — Fri, 02 Aug 2024 02:27:56 GMT

When writing a pyspark dataframe to a file, it will always write to a part file by default. This is because of partitions, even if there is only 1 partitions.

To write into a single file you can convert the pyspark dataframe to a pandas dataframe and then write to target like so.

df.toPandas().to_csv(file_path, header = True, index = False)

You should be careful when dealing with very large files because when you convert to pandas, all the data from all nodes is brought to the driver so you can write to a single output. If you face OOM issues, you can try increasing the size of the driver node.