topic Exporting delta table to one CSV in Data Engineering

Exporting delta table to one CSV

561064 — Fri, 01 Dec 2023 22:39:57 GMT

Process to export a delta table is taking ~2hrs.

Delta table has 66 partitions with total size of ~6gb, 4million rows and 270 columns.

Used below command

df.coalesce(1).write.csv("path")

what are my options to reduce the time?

Re: Exporting delta table to one CSV

Dribka — Sat, 02 Dec 2023 22:02:02 GMT

A very interesting task in front of you.... let me know how you solve it!

Re: Exporting delta table to one CSV

561064 — Mon, 04 Dec 2023 19:07:34 GMT

Hi Kainz,

None of the options I tried helped as the challenge is not reading but writing it to a one CSV file. df.repartition(numFiles).write.csv("path") has consumed the same amount of time as 'df.coalesce(1).write.csv("path")' in my case.

any other options I can explore?