topic Re: How to change line separator for csv file exported from dataframe in databricks in Data Engineering

How to change line separator for csv file exported from dataframe in databricks

tunguyen90 — Fri, 08 Nov 2019 15:41:42 GMT

Hello,

Currently, I'm facing problem with line separator inside csv file, which is exported from data frame in Azure Databricks (version Spark 2.4.3) to Azure Blob storage. All those csv files contains LF as line-separator. I need to have CRLF (\r\n) as line separator in those csv files. Although I've tried different ways to change that default line separator into my target line separator, it doesn't work. Up to now, I tried following ways

1. In databricks notebook, I added option to customize line separator as following.

df.write.format("com.databricks.spark.csv").option("header","true").option("lineSep",'\r\n').mode("overwrite").save(target_folder);

Looks like such option("lineSep") doesn't work yet, at least in my version Spark 2.4.3. I checked in other similar question, in forum they said that this doesn't work as well in Spark 2.4.4.

2. In databricks notebook, I tried to add following line

spark._jvm.java.lang.System.setProperty("line.separator", "\r\n");

Looks like that such setProperty didn't change also the default line separator once parsing and writing data to output in csv files in Azure blob storage.

Maybe someone has other suggestion / solution for this problem ? Thanks a lot for your help.

Re: How to change line separator for csv file exported from dataframe in databricks

shyam_9 — Wed, 13 Nov 2019 07:39:43 GMT

Hi @tunguyen90,

Please go through the below answer,

https://forums.databricks.com/questions/12831/custom-line-separator.html

Re: How to change line separator for csv file exported from dataframe in databricks

neilbest-db — Thu, 28 May 2020 17:35:38 GMT

The linked answer seems to be about reading. @tunguyen90 asked about writing. Aren't they distinct issues? I am also looking for latest info on the writer.

Re: How to change line separator for csv file exported from dataframe in databricks

Nikhila — Thu, 24 Dec 2020 13:02:26 GMT

Hi,

Have you got the solution for above problem.Kindly let me know.