This way

it was working ...

val df = Seq((1, 10), (2, 20), (3, 30)).toDS.toDF("sex", "date") df.show(false) // save it

df.repartition(1).write.format("parquet").mode("overwrite").save(".../temp")

// read back again val df1 = spark.read.format("parquet").load(".../temp") val df2 = df1.withColumn("cleanup" , lit("Quick silver want to cleanup"))

// BELOW 2 ARE IMPORTANT STEPS LIKE

cache

show

df2.cache // cache to avoid FileNotFoundException df2.show(2) // light action or println(df2.count) also fine

df2.repartition(1).write.format("parquet").mode("overwrite").save(".../temp")

df2.show(false)

will work for sure only thing is you need to cache and perform small action like show(1) this is altermative route thought of proposing to the users who mandatorily need it.

Result ... file : read and overwritten