topic Impact of VACUUM and retention settings on Delta Lake in Data Engineering

Impact of VACUUM and retention settings on Delta Lake

mjedy78 — Tue, 08 Jul 2025 13:28:32 GMT

I have a table that needs to support time travel for up to 6 months. To preserve the necessary metadata and data files, I’ve already configured the table with the following properties:

ALTER TABLE table_x SET TBLPROPERTIES ( 'delta.logRetentionDuration' = 'interval 180 days', 'delta.deletedFileRetentionDuration' = 'interval 180 days' );

However, there are delta lake maintanance job which runs

VACUUM table_x RETAIN 720 HOURS; -- 30 days

Will running VACUUM table_x RETAIN 720 HOURS ignore the 180-day table properties and potentially delete files needed for time travel?

Thanks

Re: Impact of VACUUM and retention settings on Delta Lake

intuz — Wed, 09 Jul 2025 06:26:49 GMT

Yes, the VACUUM table_x RETAIN 720 HOURS; command will indeed override your table-level retention properties and potentially compromise your 6-month time travel capability.
When you explicitly specify a retention period in the VACUUM command, it takes precedence over the table properties 'delta.deletedFileRetentionDuration' and 'delta.logRetentionDuration' .

In your case, the maintenance job is enforcing a 30-day retention policy that conflicts with your 180-day table configuration. The vacuum operation will remove data files older than 30 days that may still be referenced by transaction logs within your 180-day retention window.
Hope this helps!