topic Re: UTF-8 troubles in DLT in Get Started Discussions

UTF-8 troubles in DLT

EirikMa — Mon, 25 Mar 2024 10:11:12 GMT

Issues with UTF-8 in DLT

I am having issues with UTF-8 in DLT:

I have tried to set the spark config on the cluster running the DLT pipeline:

I have fixed this with normal compute under advanced settings like this:
spark.conf.set("spark.driver.extraJavaOptions", "-Dfile.encoding=UTF-8")

spark.conf.set("spark.executor.extraJavaOptions", "-Dfile.encoding=UTF-8")

However, this does not work with DLT. Have any of you guys figured this out?

- Eirik

Re: UTF-8 troubles in DLT

EirikMa — Fri, 03 May 2024 07:42:18 GMT

Hi @Retired_mod!

Sorry for a long wait...

The problem is not the columns or the data itself, the UTF-8 option for csv is working fine. The issue is with table_names not being compatible it seems. If I run the query through Auto Loader outside DLT and use backticks for catalog_name, schema_name and table_name, likt this: `dev`.`bronze`.`bokføring` it works perfectly.

Is there anyway that this can be done in DLT? Do you know the timeline when the runtime will be upgraded so that it will work?

Re: UTF-8 troubles in DLT

EirikMA1 — Mon, 02 Sep 2024 09:41:11 GMT

@Retired_mod Hi, DLT has updated its runtime but I get a different error now:

this is my code: