topic Re: How to handle multilines coming from CSV file in a quoted string in Data Engineering

How to handle multilines coming from CSV file in a quoted string

MounicaVemulapa — Wed, 12 Jun 2019 05:14:06 GMT

How to handle multilines coming from CSV file in a quoted string

Re: How to handle multilines coming from CSV file in a quoted string

mathan_pillai — Wed, 12 Jun 2019 11:17:02 GMT

Hi @Mounica Vemulapalli

Do you mean how to handle multilines in the source csv file? While using spark.read API, did you try including the multiline option set to true? please try and let us know how it goes


.option("multiLine","true")

Thanks

Re: How to handle multilines coming from CSV file in a quoted string

MounicaVemulapa — Wed, 12 Jun 2019 11:47:09 GMT

@Mathan Pillai .. Yes I tried it.. But in the file, multiline of a column is considering as row itself

Re: How to handle multilines coming from CSV file in a quoted string

mathan_pillai — Thu, 13 Jun 2019 10:07:17 GMT

Hi,

Can you try escape parameter & quote parameter to indicate which characters need to be ignored. The escape character within the quotes will be ignored. you can specify the newline character, that it needs to be ignored. please refer to below documentation for more info

https://docs.databricks.com/spark/latest/data-sources/read-csv.html#reading-files

```
quote
```
: by default the quote character is
```
"
```
, but can be set to any character. Delimiters inside quotes are ignored.
```
escape
```
: by default the escape character is
```
\
```
, but can be set to any character. Escaped quote characters are ignored.

Thanks

Re: How to handle multilines coming from CSV file in a quoted string

dataengineerfro — Fri, 15 Sep 2023 11:42:32 GMT

In my case all three options are not working. still I am facing issue data is not properly separated

escape

.option("multiLine","true")

quote