topic Re: Show Vacuum operation result (files deleted) without DRY RUN in Data Engineering

Show Vacuum operation result (files deleted) without DRY RUN

alejandrofm — Mon, 07 Mar 2022 01:09:33 GMT

Hi, I'm runing some scheduled vacuum jobs and would like to know how many files were deleted without making all the computation twice, with and without DRY RUN, is there a way to accomplish this?

Thanks!

Re: Show Vacuum operation result (files deleted) without DRY RUN

RKNutalapati — Mon, 07 Mar 2022 05:51:23 GMT

Hi @Alejandro Martinez :

I don't think we have any such command to get the statistics before vacuum and after vacuum.

Atleast I haven't come across any.

If you want to capture more details, may be you can write a function to capture the statistics as below.

Data files size:

Data files count:

Before:

var getDataFileSize = 0

val getDataFileCount = dbutils.fs.ls(<Your Table Path>").toList.size

dbutils.fs.ls(<Your Table Path>)

.foreach

{

file =>

getDataFileSize = getDataFileSize + file.size

}

After:

Repeat above

Lets see if other community members have better ideas on this.

Re: Show Vacuum operation result (files deleted) without DRY RUN

Hubert-Dudek — Mon, 07 Mar 2022 10:44:07 GMT

SELECT * FROM (DESCRIBE HISTORY table)x WHERE operation IN ('VACUUM END', 'VACUUM START');

that gives us required information:

Re: Show Vacuum operation result (files deleted) without DRY RUN

alejandrofm — Mon, 07 Mar 2022 14:13:20 GMT

Thank you! Not the solution I was looking for, but it seems nothing better exists...yet so going for that.

Thanks!!!

Re: Show Vacuum operation result (files deleted) without DRY RUN

RKNutalapati — Mon, 07 Mar 2022 14:22:17 GMT

We have to enable logging to capture the logs for vacuum.

spark.conf.set("spark.databricks.delta.vacuum.logging.enabled","true")