cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Seeing history even after vacuuming the Delta table

Kavi_007
New Contributor III

Hi,

I'm trying to do the vacuum on a Delta table within a unity catalog. The default retention is 7 days. Though I vacuum the table, I'm able to see the history beyond 7 days. Tried restarting the cluster but still not working. What would be the fix ? 

1 ACCEPTED SOLUTION

Accepted Solutions

Lakshay
Esteemed Contributor
Esteemed Contributor

Hi @Kavi_007 , Vacuum only deletes stale files i.e. the files that are not being tracked by delta log. If you perform a delete operation, then the files will become stale and will be cleared after 7 days. From the delta history of your table, I do not see any delete operation. So, vacuum will not delete any file.

View solution in original post

7 REPLIES 7

Kaniz_Fatma
Community Manager
Community Manager

Hi @Kavi_007

  • Verify that the VACUUM command is indeed running on the correct table.
  • Check if there are any custom retention settings or overrides in your environment.

Kavi_007
New Contributor III

Yes, below are the commands. 

 

VACUUM `dev-sales-catalog`.silver.orders   -- This does not do vaccum
 
Kavi_007_0-1712201764703.png
SELECT * FROM `dev-sales-catalog`.silver.orders VERSION AS OF 0        --  This still does bring the results with the version which is 10 days older.
 0Kavi_007_1-1712201816573.png

History of the table:

Kavi_007_3-1712201997933.png

 


 


 

 
 

Kavi_007
New Contributor III

 @Kaniz_Fatma  - Could you please check this? 

Lakshay
Esteemed Contributor
Esteemed Contributor

Hi @Kavi_007 , Vacuum only deletes stale files i.e. the files that are not being tracked by delta log. If you perform a delete operation, then the files will become stale and will be cleared after 7 days. From the delta history of your table, I do not see any delete operation. So, vacuum will not delete any file.

Kavi_007
New Contributor III

@Lakshay - you are right. I performed a couple of delete statements and then did VACCUM. It worked now. Thanks for your help !

Lakshay
Esteemed Contributor
Esteemed Contributor

Happy to help!

Kavi_007
New Contributor III

No, that's wrong. VACUUM removes all files from the table directory that are not managed by Delta, as well as data files that are no longer in the latest state of the transaction log for the table and are older than a retention threshold.

VACUUM - Azure Databricks - Databricks SQL | Microsoft Learn

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!