- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-16-2022 10:37 PM
When writing a dataframe in Pyspark to a CSV file, a folder is created and a partitioned CSV file is created. I have then rename this file in order to distribute it my end user.
Is there any way I can simply write my data to a CSV file, with the name I specified, and have that single file in the folder I specified ?
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-16-2022 11:26 PM
it will always write to a folder due to the parallel nature of spark.
If that is an issue, you can use magic command %sh to move the .csv file a level up and also rename it.
So use the 'mv' command.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-16-2022 11:13 PM
yes, but you have to do a coalesce(1). This will generate a single csv file, however you will also lose some parallelism as this coalesce(1) is propagated upstream.
Also do not forget to disable the writing of _SUCCESS etc files (see this topic)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-16-2022 11:20 PM
Thanks Werners. however it still writes to a folder, and I still need to rename the file, and copy it out etc.
I would like test1.csv to be a file in the root folder. Not a folder.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-16-2022 11:26 PM
it will always write to a folder due to the parallel nature of spark.
If that is an issue, you can use magic command %sh to move the .csv file a level up and also rename it.
So use the 'mv' command.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-08-2022 01:56 AM
The csv file will have random name, can you show me how you will rename it without going into hassel of copying its name?
For example lets say name of root folder is Main, inside main i wrote csv using coalsce(1) and the structure is Main/data.csv/RandomBigName-part-00000xyz.csv
Now i want to move csv file inside Main folder and lets say name it as dummyData.csv... So final structure which i want is Main/dummyData.csv
Please help
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-09-2024 09:41 AM
Could you please provide an example of using %sh or mv to move and rename the csv?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-16-2022 11:28 PM
Thanks for confirming that that's the only way 🙂
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-04-2024 10:36 AM - edited 10-04-2024 10:41 AM
I know this post is a little old, but Chat GPT actually put together a very clean and straightforward solution for me (in scala):