Databricks not able to preserve special character while writing data

[Priya_Data_Eng]

I am trying to write data in csv format 

(df. coalesce(1) write

• format("csv")
• option ("header","true")
• Option (" sep","|시")
• option("encoding", "UTF-8”)
• mode ("overwrite")
• save("/mnt/Filestore/data/job1))Now here the problem is - there is one column in this data frame df with some value starting with a ? Enclosed in square box.we can read this data from source query in this df but after writing using above code , that special character is missing in csv file. Even if I read data in databricks, this special character is missing.could you please suggest what can be the solution here to preserve this character 

[Kaniz_Fatma]

Hi @Priya_Data_Eng,

1. Since the pipe character (|) is part of the special character, it may be causing issues. Try using a different separator character that is not part of the special character, such as a comma (,) or a tab character (\t).
2. If the UTF-8 encoding is not preserving the special character, try using a different encoding, such as UTF-16 or UTF-32.
3. If the CSV format is not preserving the special character, try using a different file format that is better suited for handling special characters, such as Parquet or ORC.
4. If the special character is still not preserved, you can try using a custom character escape sequence to represent the special character in the CSV file. For example, you can replace the special character with a unique string that represents it, such as "$$?$$".

Try implementing these suggestions, and let me know if you need further assistance! 😊