Writing part files in single text file
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-01-2024 06:28 AM
i want to write all my part file into a single text file is there anything i can do
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-01-2024 06:49 AM
coalesce with one partition might be your friend:
(
df
.coalesce(1)
.write.format('csv')
.option('header', 'true')
.save('one-file.csv')
)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-01-2024 07:27 PM
When writing a pyspark dataframe to a file, it will always write to a part file by default. This is because of partitions, even if there is only 1 partitions.
To write into a single file you can convert the pyspark dataframe to a pandas dataframe and then write to target like so.
df.toPandas().to_csv(file_path, header = True, index = False)
You should be careful when dealing with very large files because when you convert to pandas, all the data from all nodes is brought to the driver so you can write to a single output. If you face OOM issues, you can try increasing the size of the driver node.

