topic How do I prevent _success and _committed files in my write output? in Data Engineering

How do I prevent _success and _committed files in my write output?

PradeepRavi — Thu, 02 Aug 2018 04:36:24 GMT

Is there a way to prevent the _success and _committed files in my output. It's a tedious task to navigate to all the partitions and delete the files.

Note : Final output is stored in Azure ADLS

Re: How do I prevent _success and _committed files in my write output?

AndrewSears — Fri, 03 Aug 2018 11:15:44 GMT

This was recommended on StackOverflow though I haven't tested with ADLS yet.

sc._jsc.hadoopConfiguration().set("mapreduce.fileoutputcommitter.marksuccessfuljobs", "false")

Note it may impact the whole cluster.

You could also use dbutils.fs.rm step to remove any created files.

cheers,

Andrew

Re: How do I prevent _success and _committed files in my write output?

PradeepRavi — Tue, 07 Aug 2018 11:30:03 GMT

This solution is working in my local intellij setup but not with Databricks notebook setup.

Re: How do I prevent _success and _committed files in my write output?

AndrewSears — Tue, 07 Aug 2018 15:46:53 GMT

Did you try with a new Databricks cluster using initialization scripts?

https://docs.databricks.com/user-guide/clusters/init-scripts.html

Re: How do I prevent _success and _committed files in my write output?

DD_Sharma — Fri, 24 Jan 2020 12:53:29 GMT

A combination of below three properties will help to disable writing all the transactional files which start with "_".

We can disable the transaction logs of spark parquet write using "spark.sql.sources.commitProtocolClass = org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol". This will help to disable the "committed<TID>" and "started<TID>" files but still _SUCCESS, _common_metadata and _metadata files will generate.
We can disable the _common_metadata and _metadata files using "parquet.enable.summary-metadata=false".
We can also disable the _SUCCESS file using "mapreduce.fileoutputcommitter.marksuccessfuljobs=false".

Re: How do I prevent _success and _committed files in my write output?

DebashisPaul1 — Wed, 13 May 2020 02:22:28 GMT

This is very helpful ... Thanks for the information ... Just to add more info to it if somebody wants to disable it at Cluster level for Spark 2.4.5. they can edit the Spark Cluster -) Advanced Options and add above but you need to use <variable> <value> like below :

parquet.enable.summary-metadata false

If you want to add it in databricks notebook you can do like this:

spark.conf.set("parquet.enable.summary-metadata", "false")

Re: How do I prevent _success and _committed files in my write output?

shan_chandra — Sat, 04 Jun 2022 18:57:58 GMT

Please find the below steps to remove _SUCCESS, _committed and _started files.

spark.conf.set("spark.databricks.io.directoryCommit.createSuccessFile","false") to remove success file.
run vacuum command multiple times until _committed and _started files are removed.

spark.sql("VACUUM '<file-location>' RETAIN 0 HOURS")