topic Re: Seeing history even after vacuuming the Delta table in Data Engineering

Seeing history even after vacuuming the Delta table

Kavi_007 — Mon, 01 Apr 2024 19:44:21 GMT

Hi,

I'm trying to do the vacuum on a Delta table within a unity catalog. The default retention is 7 days. Though I vacuum the table, I'm able to see the history beyond 7 days. Tried restarting the cluster but still not working. What would be the fix ?

Re: Seeing history even after vacuuming the Delta table

Kavi_007 — Thu, 04 Apr 2024 03:40:12 GMT

Yes, below are the commands.

VACUUM `dev-sales-catalog`.silver.orders -- This does not do vaccum

SELECT * FROM `dev-sales-catalog`.silver.orders VERSION AS OF 0 -- This still does bring the results with the version which is 10 days older.

History of the table:

Re: Seeing history even after vacuuming the Delta table

Kavi_007 — Thu, 04 Apr 2024 14:06:55 GMT

@Retired_mod - Could you please check this?

Re: Seeing history even after vacuuming the Delta table

Lakshay — Thu, 04 Apr 2024 14:54:41 GMT

Hi @Kavi_007 , Vacuum only deletes stale files i.e. the files that are not being tracked by delta log. If you perform a delete operation, then the files will become stale and will be cleared after 7 days. From the delta history of your table, I do not see any delete operation. So, vacuum will not delete any file.

Re: Seeing history even after vacuuming the Delta table

Kavi_007 — Thu, 04 Apr 2024 19:06:25 GMT

No, that's wrong. VACUUM removes all files from the table directory that are not managed by Delta, as well as data files that are no longer in the latest state of the transaction log for the table and are older than a retention threshold.

VACUUM - Azure Databricks - Databricks SQL | Microsoft Learn

Re: Seeing history even after vacuuming the Delta table

Kavi_007 — Thu, 04 Apr 2024 20:00:17 GMT

@Lakshay - you are right. I performed a couple of delete statements and then did VACCUM. It worked now. Thanks for your help !

Re: Seeing history even after vacuuming the Delta table

Lakshay — Fri, 05 Apr 2024 16:38:31 GMT

Happy to help!