Hello,
Currently, I'm facing problem with line separator inside csv file, which is exported from data frame in Azure Databricks (version Spark 2.4.3) to Azure Blob storage. All those csv files contains LF as line-separator. I need to have CRLF (\r\n) as line separator in those csv files. Although I've tried different ways to change that default line separator into my target line separator, it doesn't work. Up to now, I tried following ways
1. In databricks notebook, I added option to customize line separator as following.
df.write.format("com.databricks.spark.csv").option("header","true").option("lineSep",'\r\n').mode("overwrite").save(target_folder);
Looks like such option("lineSep") doesn't work yet, at least in my version Spark 2.4.3. I checked in other similar question, in forum they said that this doesn't work as well in Spark 2.4.4.
2. In databricks notebook, I tried to add following line
spark._jvm.java.lang.System.setProperty("line.separator", "\r\n");
Looks like that such setProperty didn't change also the default line separator once parsing and writing data to output in csv files in Azure blob storage.
Maybe someone has other suggestion / solution for this problem ? Thanks a lot for your help.
