topic Re: An unidentified special character is added in outbound file when transformed in databricks. Please help with suggestion? in Data Engineering

An unidentified special character is added in outbound file when transformed in databricks. Please help with suggestion?

JK2021 — Mon, 18 Oct 2021 13:49:00 GMT

Data from external source is copied to ADLS, which further gets picked up by databricks, then this massaged data is put in the outbound file . A special character ? (question mark in black diamond) is seen in some fields in outbound file which may break existing code is not identified.

Re: An unidentified special character is added in outbound file when transformed in databricks. Please help with suggestion?

Prabakar — Mon, 18 Oct 2021 14:03:38 GMT

Hi @Jazmine Kochan , what type of data is being copied? Does the data have any Unicode characters or symbols like ç ã,...?

Re: An unidentified special character is added in outbound file when transformed in databricks. Please help with suggestion?

JK2021 — Mon, 18 Oct 2021 14:28:52 GMT

Hi Prabakar,

Thanks for promt response.

It is a text file with customer data.

I have not seen such characters in the data but in text entry fields, this kind of data could be entered by client.

Re: An unidentified special character is added in outbound file when transformed in databricks. Please help with suggestion?

JK2021 — Mon, 18 Oct 2021 14:44:54 GMT

So yes, text could contain such characters.

Re: An unidentified special character is added in outbound file when transformed in databricks. Please help with suggestion?

Prabakar — Mon, 18 Oct 2021 14:51:12 GMT

So the cause of the issue is those Unicode characters. I believe there should be a fix for this. I shall check and get back here.

Re: An unidentified special character is added in outbound file when transformed in databricks. Please help with suggestion?

JK2021 — Mon, 18 Oct 2021 14:58:32 GMT

Thanks much!

Re: An unidentified special character is added in outbound file when transformed in databricks. Please help with suggestion?

-werners- — Mon, 18 Oct 2021 15:04:44 GMT

Are you sure it is Databricks which puts the special character in place?

It could also have happened during the copy of the external system to ADLS.

If you use Azure Data Factory f.e. you have to define the encoding (UTF-8 or UTF-16, ...)

Re: An unidentified special character is added in outbound file when transformed in databricks. Please help with suggestion?

JK2021 — Mon, 18 Oct 2021 15:15:37 GMT

Yes we checked all the files in the flow. It is output file from Databricks in which question mark character is seen at beginning of some lines in text fields.

Re: An unidentified special character is added in outbound file when transformed in databricks. Please help with suggestion?

JK2021 — Mon, 18 Oct 2021 15:29:54 GMT

Hi Prabakar

Could it be developer's code - which could be adding this special character?

Re: An unidentified special character is added in outbound file when transformed in databricks. Please help with suggestion?

Prabakar — Mon, 18 Oct 2021 15:38:18 GMT

This needs encoding. you can try encoding the output while reading the file.

.option("encoding", "UTF-16LE")

Please refer to the below:

https://docs.microsoft.com/en-us/azure/databricks/kb/data-sources/json-unicode

https://community.databricks.com/s/question/0D53f00001HKHnfCAH/issues-with-utf16-files-and-unicode-characters

Re: An unidentified special character is added in outbound file when transformed in databricks. Please help with suggestion?

JK2021 — Wed, 10 Nov 2021 21:30:29 GMT

Do i need to encode and decode too?? Currently incorrect data is displayed @Prabakar Ammeappin