Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-14-2022 09:48 PM
val spark:SparkSession = SparkSession.builder()
.master("local[3]")
.appName("SparkByExamples.com")
.getOrCreate()
//Spark Read CSV File
val df = spark.read.option("header",true).csv("address.csv")
//Write DataFrame to address directory
df.write.csv("address")
Above write statement writes a 3 CSV files and .CRC and _SUCCESS files.
Is there any option in Spark not to write these files? I found an article that explains how to remove these files after writing https://sparkbyexamples.com/spark/spark-write-dataframe-single-csv-file/ but I can't use this for several reasons.
Hope the question is clear and looking forward some answer here.
Appreciate.
Labels:
- Labels:
-
Azure