aladda
Databricks Employee
Databricks Employee

You get multiple files in a folder because spark writes each shuffle partition in-place out to a "part..." file to avoid network I/O. You can use coalesce to bring all the shuffles into a single partition and write it out to a single file but be mindful of the performance implications

df.coalesce(1)
   .write.format(csv)
   .option("header", "true")
   .save("singlefile.csv")

View solution in original post