Spark adding NUL
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-28-2021 07:39 PM
Hi I have a DF which contains Json string so the value is like {"key": Value, "anotherKey": anotherValue}, so when I am trying to write the DF containing this string to the CSV, spark is adding NUL character af the front of this line and at the end, so the final line is like
NUL{"key": Value, "anotherKey": anotherValue}NUL
I really don't want this to happen, how can I prevent this?
The code I am using is
df.coalesce(1).write.format("csv").option("header", false).option("quote", "").save(path)