topic Re: Spark adding NUL in Data Engineering

Spark adding NUL

vasu_sethia — Fri, 29 Oct 2021 02:39:15 GMT

Hi I have a DF which contains Json string so the value is like {"key": Value, "anotherKey": anotherValue}, so when I am trying to write the DF containing this string to the CSV, spark is adding NUL character af the front of this line and at the end, so the final line is like

NUL{"key": Value, "anotherKey": anotherValue}NUL

I really don't want this to happen, how can I prevent this?

The code I am using is

df.coalesce(1).write.format("csv").option("header", false).option("quote", "").save(path)

Re: Spark adding NUL

Piper_Wilson — Fri, 29 Oct 2021 03:01:52 GMT

Hello, @Vasu Sethia! My name is Piper and I'm one of the moderators for Databricks. Welcome and thank you for your question. Let's give it a bit longer to see what the community has to say. Otherwise, we'll circle back around soon.

Re: Spark adding NUL

-werners- — Fri, 29 Oct 2021 08:14:53 GMT

Are you writing the actual json string in a csv, or do you flatten the json into a table structure and write that to csv?

Re: Spark adding NUL

vasu_sethia — Fri, 29 Oct 2021 09:09:27 GMT

I have a value in my dataframe column in the format of Json string, I am trying to write the dataframe to the csv

_________

Value

__________

{"Name": ABC, "age": 12}

__________

Re: Spark adding NUL

-werners- — Fri, 29 Oct 2021 11:30:52 GMT

Hard to tell without having the code, but it might be the separator for the csv? You do have comma's in the string, and comma is the default separator for csv.

Re: Spark adding NUL

vasu_sethia — Fri, 29 Oct 2021 12:10:53 GMT

df.coalesce(1).write.format("csv").option("header", false).option("quote", "").save(path)

This is the code and yes I do have comma in the string

Re: Spark adding NUL

-werners- — Fri, 29 Oct 2021 13:41:15 GMT

I mean the code for 'df'.

Can you try to write with option("sep", ";")?

Re: Spark adding NUL

vasu_sethia — Fri, 29 Oct 2021 15:11:40 GMT

Thank you so much, this worked for me

Re: Spark adding NUL

jose_gonzalez — Fri, 29 Oct 2021 23:04:28 GMT

hi @Vasu Sethia ,

If Werners' response fully answered your question, would you be happy to mark the answer as best so that others can quickly find the solution?