topic Re: How do I get the size of files cleaned up by a vacuum for a Delta table. in Data Engineering

How do I get the size of files cleaned up by a vacuum for a Delta table.

User16826987838 — Fri, 18 Jun 2021 21:07:31 GMT

Re: How do I get the size of files cleaned up by a vacuum for a Delta table.

Ryan_Chynoweth — Mon, 21 Jun 2021 18:15:26 GMT

The output of the optimized command produces the following metrics:

Number of Files Added
Number of Files Removed
min, max, avg, total files, and total size of files added
min, max, avg, total files, and total size of files removed
number of partitions optimized
z order stats
number of batches
total files considered
total files skipped

If that information does not provide the details required then you would need to scan the file system before and after running the command to collect and analyze the data yourself.

Re: How do I get the size of files cleaned up by a vacuum for a Delta table.

sajith_appukutt — Mon, 21 Jun 2021 21:06:16 GMT

def getVaccumSize(table: String): Long = {
  val listFiles = spark.sql(s"VACUUM $table DRY RUN").select("path").collect().map(_(0)).toList
  var sum = 0L
  listFiles.foreach(x => sum += dbutils.fs.ls(x.toString)(0).size)
  sum 
}
 
getVaccumSize("<your-table-name>")

You could use this function to get the size of files cleaned up