cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Trying to vacuum a table that is constantly being "createdOrReplaced"

alonisser
Contributor

and it seems that older data (From the "replaced" table) isn't being removed, long after the retention period

 

I'd be glad for clues on how to handle this

2 REPLIES 2

swethaNandan
Databricks Employee
Databricks Employee

Eligibility of a file for vacuum is based on the date that the file was ,asked for deletion and not on the creation timestamp of the file.
Delta table data files are deleted according to the time they have been logically removed from Deltaโ€™s transaction log plus retention hours, not their modification timestamps on the storage system.

Can you please confirm that the files were marked for removal before the retention period to make sure that they are eligible.

alonisser
Contributor

Thanks, I know that, but the table history shows 30 days, but the actual data size and number of files and all other indicators , correlate to 170 days. 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group