I'm running Spark 2.2.0 at the moment. Currently I'm facing an issue when importing data of Mexican origin, where the characters can have special characters and with multiline for certain columns.

Ideally, this is the command I'd like to run:

T_new_exp = spark.read\   
.option("charset", "ISO-8859-1")\   
.option("parserLib", "univocity")\
.option("multiLine", "true")\   
.schema(schema)\
.csv(file)

However, using the above gives me properly lined rows but without the correct charset. Instead of displaying e acute for example, I'm getting the replacement character (U+FFFD). It's only when I remove the multiline option do I get the right charset (but without the multiline issue being fix).

The only solution that I have to workaround this problem for now is to preprocess the data separately before it is loaded to databricks; that is - fix the multiline first in unix and let Databricks handle the unicode issues later.

Is there a simpler way than this?