Issues with UTF-8 in DLT
I am having issues with UTF-8 in DLT:
I have tried to set the spark config on the cluster running the DLT pipeline:
I have fixed this with normal compute under advanced settings like this:
spark.conf.set("spark.driver.extraJavaOptions", "-Dfile.encoding=UTF-8")
spark.conf.set("spark.executor.extraJavaOptions", "-Dfile.encoding=UTF-8")
However, this does not work with DLT. Have any of you guys figured this out?
- Eirik