Seeing history even after vacuuming the Delta table

[Kavi_007]

Hi,

I'm trying to do the vacuum on a Delta table within a unity catalog. The default retention is 7 days. Though I vacuum the table, I'm able to see the history beyond 7 days. Tried restarting the cluster but still not working. What would be the fix ? 

[Kaniz]

Hi @Kavi_007, 

• Verify that the VACUUM command is indeed running on the correct table.
• Check if there are any custom retention settings or overrides in your environment.

[Kavi_007]

Yes, below are the commands. 

VACUUM `dev-sales-catalog`.silver.orders   -- This does not do vaccum

SELECT * FROM `dev-sales-catalog`.silver.orders VERSION AS OF 0        --  This still does bring the results with the version which is 10 days older.

 0

History of the table:

[Kavi_007]

 @Kaniz  - Could you please check this? 

[Lakshay · Accepted solution]

Hi @Kavi_007 , Vacuum only deletes stale files i.e. the files that are not being tracked by delta log. If you perform a delete operation, then the files will become stale and will be cleared after 7 days. From the delta history of your table, I do not see any delete operation. So, vacuum will not delete any file.

[Kavi_007]

@Lakshay - you are right. I performed a couple of delete statements and then did VACCUM. It worked now. Thanks for your help !

[Lakshay]

Happy to help!

[Kavi_007]

No, that's wrong. VACUUM removes all files from the table directory that are not managed by Delta, as well as data files that are no longer in the latest state of the transaction log for the table and are older than a retention threshold.

VACUUM - Azure Databricks - Databricks SQL | Microsoft Learn