Writing part files in single text file

[Manthansingh]

i want to write all my part file into a single text file is there anything i can do 

[Witold]

coalesce with one partition might be your friend:

(
  df
   .coalesce(1)
   .write.format('csv')
   .option('header', 'true')
   .save('one-file.csv')
)

[Edthehead]

When writing a pyspark dataframe to a file, it will always write to a part file by default. This is because of partitions, even if there is only 1 partitions.

To write into a single file you can convert the pyspark dataframe to a pandas dataframe and then write to target like so.

df.toPandas().to_csv(file_path, header = True, index = False)

You should be careful when dealing with very large files because when you convert to pandas, all the data from all nodes is brought to the driver so you can write to a single output. If you face OOM issues, you can try increasing the size of the driver node.