Show Vacuum operation result (files deleted) without DRY RUN

[alejandrofm]

Hi, I'm runing some scheduled vacuum jobs and would like to know how many files were deleted without making all the computation twice, with and without DRY RUN, is there a way to accomplish this?

Thanks!

[RKNutalapati]

Hi @Alejandro Martinez​ :

 I don't think we have any such command to get the statistics before vacuum and after vacuum.

 Atleast I haven't come across any.

 If you want to capture more details, may be you can write a function to capture the statistics as below.

 Data files size:

 Data files count:

 Before:

var getDataFileSize = 0

val getDataFileCount = dbutils.fs.ls(<Your Table Path>").toList.size

dbutils.fs.ls(<Your Table Path>)

 .foreach

 {

  file =>

  getDataFileSize = getDataFileSize + file.size

 }

 After:

   Repeat above

Lets see if other community members have better ideas on this.

[Hubert-Dudek · Accepted solution]

SELECT * FROM (DESCRIBE HISTORY table)x WHERE operation IN ('VACUUM END', 'VACUUM START');

that gives us required information:

[alejandrofm]

Thank you! Not the solution I was looking for, but it seems nothing better exists...yet so going for that.

Thanks!!!

[RKNutalapati]

We have to enable logging to capture the logs for vacuum.

spark.conf.set("spark.databricks.delta.vacuum.logging.enabled","true")