writing spark dataframe as CSV to a repo
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-12-2024 11:25 PM
Hi,
I wrote a spark dataframe as csv to a repo (synced with github). But when I checked the folder, the file wasn't there.
Here is my code:
spark_df.write.format('csv').option('header','true').mode('overwrite').save('/Repos/abcd/mno/data')
No error message ! Nevertheless no file inside the folder 'data'. In addition 'abcd' is my repo name, inside it is a folder 'mno' which contains 'data'.
-Thx
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-13-2024 02:44 AM
Hi @Retired_mod ,
Thanks for the help. But my problem is that I do not see any files in the target folder. I am looking for options.
-Regards
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-13-2024 03:52 AM
the folder 'Repos' is not your repo, it's `dbfs:/Repos`, please check
dbutils.fs.ls('/Repos/abcd/mno/data')
