Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-22-2021 08:13 PM
You get multiple files in a folder because spark writes each shuffle partition in-place out to a "part..." file to avoid network I/O. You can use coalesce to bring all the shuffles into a single partition and write it out to a single file but be mindful of the performance implications
df.coalesce(1)
.write.format(csv)
.option("header", "true")
.save("singlefile.csv")