Databricks not able to preserve special character while writing data

Priya_Data_Eng
New Contributor

I am trying to write data in csv format 

(df. coalesce(1) write

  • format("csv")
  • option ("header","true")
  • Option (" sep","|시")
  • option("encoding", "UTF-8”)
  • mode ("overwrite")
  • save("/mnt/Filestore/data/job1))Now here the problem is - there is one column in this data frame df with some value starting with a ? Enclosed in square box.we can read this data from source query in this df but after writing using above code , that special character is missing in csv file. Even if I read data in databricks, this special character is missing.could you please suggest what can be the solution here to preserve this character