DD_Sharma
Databricks Employee
Databricks Employee

A combination of below three properties will help to disable writing all the transactional files which start with "_".

  1. We can disable the transaction logs of spark parquet write using "spark.sql.sources.commitProtocolClass = org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol". This will help to disable the "committed<TID>" and "started<TID>" files but still _SUCCESS, _common_metadata and _metadata files will generate.
  2. We can disable the _common_metadata and _metadata files using "parquet.enable.summary-metadata=false".
  3. We can also disable the _SUCCESS file using "mapreduce.fileoutputcommitter.marksuccessfuljobs=false".