Exporting delta table to one CSV

[561064]

Process to export a delta table is taking ~2hrs.

Delta table has 66 partitions with total size of ~6gb, 4million rows and 270 columns.

Used below command

df.coalesce(1).write.csv("path")

what are my options to reduce the time?

[Dribka]

A very interesting task in front of you.... let me know how you solve it!

[561064]

Hi Kainz,

None of the options I tried helped as the challenge is not reading but writing it to a one CSV file. df.repartition(numFiles).write.csv("path") has consumed the same amount of time as 'df.coalesce(1).write.csv("path")' in my case.

any other options I can explore?