UTF-8 troubles in DLT

EirikMa
New Contributor II

Issues with UTF-8 in DLT

I am having issues with UTF-8 in DLT:

EirikMa_0-1711360526822.png

I have tried to set the spark config on the cluster running the DLT pipeline: 

EirikMa_1-1711361452104.png

 

I have fixed this with normal compute under advanced settings like this:
spark.conf.set("spark.driver.extraJavaOptions", "-Dfile.encoding=UTF-8")

spark.conf.set("spark.executor.extraJavaOptions", "-Dfile.encoding=UTF-8")

However, this does not work with DLT. Have any of you guys figured this out?

- Eirik