cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

UTF-8 troubles in DLT

EirikMa
New Contributor II

Issues with UTF-8 in DLT

I am having issues with UTF-8 in DLT:

EirikMa_0-1711360526822.png

I have tried to set the spark config on the cluster running the DLT pipeline: 

EirikMa_1-1711361452104.png

 

I have fixed this with normal compute under advanced settings like this:
spark.conf.set("spark.driver.extraJavaOptions", "-Dfile.encoding=UTF-8")

spark.conf.set("spark.executor.extraJavaOptions", "-Dfile.encoding=UTF-8")

However, this does not work with DLT. Have any of you guys figured this out?

- Eirik

2 REPLIES 2

EirikMa
New Contributor II

Hi @Retired_mod

Sorry for a long wait...

The problem is not the columns or the data itself, the UTF-8 option for csv is working fine. The issue is with table_names not being compatible it seems. If I run the query through Auto Loader outside DLT and use backticks for catalog_name, schema_name and table_name, likt this: `dev`.`bronze`.`bokføring` it works perfectly.

Is there anyway that this can be done in DLT? Do you know the timeline when the runtime will be upgraded so that it will work?

 

EirikMA1
New Contributor II

@Retired_mod Hi, DLT has updated its runtime but I get a different error now:

EirikMA1_0-1725269942827.png

this is my code:

EirikMA1_1-1725269998458.png

 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group