Why I still see delta history if I made a vacuum with just 5 hours retention

chevichenk — Sat, 11 Nov 2023 02:48:17 GMT

Hi, everyone!

I execute a vacuum with 5 hours retention but I can see all the history of versions, even I can query those older version of the table.

Plus, when I see the history version, it doesn't start with zero (supposed to be the creation of the table), so, if making vacuum with few hours of retention doesn't clear the history version, how can this happen? ‌‌🤔

Before vacuum, oldest version starts in 30 and ends in 83

After vacuum, oldest version starts in 30 and ends in 85 (start and end vacuum) version zero must be the creation of the table

Thanks for your attention! 🙂

Re: Why I still see delta history if I made a vacuum with just 5 hours retention

Rom — Sat, 11 Nov 2023 13:17:07 GMT

Hi,
When disk caching is enabled, a cluster might contain data from Parquet files that have been deleted with VACUUM. Therefore, it may be possible to query the data of previous table versions whose files have been deleted. Restarting the cluster will remove the cached data. https://docs.databricks.com/en/delta/vacuum.html

topic Why I still see delta history if I made a vacuum with just 5 hours retention in Data Engineering

Why I still see delta history if I made a vacuum with just 5 hours retention

Re: Why I still see delta history if I made a vacuum with just 5 hours retention