Databricks not able to preserve special character while writing data

Priya_Data_Eng — Thu, 04 Jul 2024 16:03:02 GMT

I am trying to write data in csv format

(df. coalesce(1) write

format("csv")
option ("header","true")
Option (" sep","|시")
option("encoding", "UTF-8”)
mode ("overwrite")
save("/mnt/Filestore/data/job1))Now here the problem is - there is one column in this data frame df with some value starting with a ? Enclosed in square box.we can read this data from source query in this df but after writing using above code , that special character is missing in csv file. Even if I read data in databricks, this special character is missing.could you please suggest what can be the solution here to preserve this character

topic Databricks not able to preserve special character while writing data in Data Engineering