topic Re: How to write a Spark DataFrame to CSV file with our .CRC in Azure Databricks? in Data Engineering

How to write a Spark DataFrame to CSV file with our .CRC in Azure Databricks?

prapot — Tue, 15 Feb 2022 05:48:50 GMT

val spark:SparkSession = SparkSession.builder()

.master("local[3]")

.appName("SparkByExamples.com")

.getOrCreate()

//Spark Read CSV File

val df = spark.read.option("header",true).csv("address.csv")

//Write DataFrame to address directory

df.write.csv("address")

Above write statement writes a 3 CSV files and .CRC and _SUCCESS files.

Is there any option in Spark not to write these files? I found an article that explains how to remove these files after writing https://sparkbyexamples.com/spark/spark-write-dataframe-single-csv-file/ but I can't use this for several reasons.

Hope the question is clear and looking forward some answer here.

Appreciate.

Re: How to write a Spark DataFrame to CSV file with our .CRC in Azure Databricks?

-werners- — Tue, 15 Feb 2022 07:06:57 GMT

spark.conf.set("spark.sql.sources.commitProtocolClass", "org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol")

spark.conf.set("parquet.enable.summary-metadata", "false")

spark.conf.set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false")

There parameters avoid writing any metadata files.

The fact you have multiple csv files is the result of parallel processing. If you do not want that you will have to add coalesce(1) to your write statement.

But that will impact the performance of your spark code.

Re: How to write a Spark DataFrame to CSV file with our .CRC in Azure Databricks?

Nw2this — Tue, 09 Jan 2024 02:09:53 GMT

Will your csv have the name prefix 'part-' or can you name it whatever you like?