The way to turn off the default escaping of the double quote character (") with the backslash character (\) - i.e. to avoid escaping for all characters entirely, you must add an .option() method call with just the right parameters after the .write() method call. The goal of the option() method call is to change how the csv() method "finds" instances of the "quote" character as it is emitting the content. To do this, you must change the default of what a "quote" actually means; i.e. change the character sought from being a double quote character (") to a Unicode "\u0000" character (essentially providing the Unicode NUL character assuming it won't ever occur within the document).
Here's Scala code achieving the effect. The second to last line (ending with magic is happening here) is the critical line and looks exactly the same in Python (as it does here in Scala).
val dataFrame =
spark.sql("SELECT * FROM some_table_with_odd_characters_column")
val unitEmitCsv =
dataframe
.write
.option("header", true)
.option("quote", "\u0000") //magic is happening here
.csv("/FileStore/temp.csv")
This was only one of several lessons I learned attempting to work with Apache Spark and emitting .csv files. For more information and context on this, please see the blog post I wrote titled "Example Apache Spark ETL Pipeline Integrating a SaaS".
