UTF-8 troubles in DLT

EirikMa
New Contributor II

‎03-25-2024 03:11 AM

Issues with UTF-8 in DLT

I am having issues with UTF-8 in DLT:

EirikMa_0-1711360526822.png

I have tried to set the spark config on the cluster running the DLT pipeline: 

EirikMa_1-1711361452104.png

 

I have fixed this with normal compute under advanced settings like this:
spark.conf.set("spark.driver.extraJavaOptions", "-Dfile.encoding=UTF-8")

spark.conf.set("spark.executor.extraJavaOptions", "-Dfile.encoding=UTF-8")

However, this does not work with DLT. Have any of you guys figured this out?

- Eirik

0 Kudos

EirikMa
New Contributor II

‎05-03-2024 12:42 AM

Hi @Retired_mod! 

Sorry for a long wait...

The problem is not the columns or the data itself, the UTF-8 option for csv is working fine. The issue is with table_names not being compatible it seems. If I run the query through Auto Loader outside DLT and use backticks for catalog_name, schema_name and table_name, likt this: `dev`.`bronze`.`bokføring` it works perfectly.

Is there anyway that this can be done in DLT? Do you know the timeline when the runtime will be upgraded so that it will work?

 

0 Kudos

EirikMA1
New Contributor II

‎09-02-2024 02:41 AM

@Retired_mod Hi, DLT has updated its runtime but I get a different error now:

EirikMA1_0-1725269942827.png

this is my code:

EirikMA1_1-1725269998458.png

 

0 Kudos