Special character data preservation
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-04-2024 09:09 AM - edited 07-04-2024 09:10 AM
- This data frame has two columns name and info. Name has value as John and info has vale as 1® VOC.
After writing this data, I can read correct values in databricks but when I download the csv file and load it in notepad ( utf-8 ) , it is showing no values.
even if I click on edit in container, it is showing empty values.
What can be the reason? And what is the solution ?
df. write. format (‘csv').option ('header', True).option( 'encoding', 'UTF-8').option(' sep', '|^|’) .mode ('overwrite'). save('/mnt/Filestore/data/job1/test.dta')
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-04-2024 10:39 AM
Try to read the file back into databricks using spark.read, do you the see the charchaters showing? if yes, then it is an editor problem, use another editor such as Notepad++, if not, then the data is not on the write encoder, try different encoder other than UTF-8, maybe UTF-16
