shan_chandra
Databricks Employee
Databricks Employee

‎06-04-2022 11:57 AM

Please find the below steps to remove _SUCCESS, _committed and _started files.

  1. spark.conf.set("spark.databricks.io.directoryCommit.createSuccessFile","false") to remove success file.
  2. run vacuum command multiple times until _committed and _started files are removed.
spark.sql("VACUUM '<file-location>' RETAIN 0 HOURS")