Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-31-2024 04:09 AM
It is due to the retention Period for Change Data Feed: When CDF is enabled, Databricks retains the change data for a specified period. This retention period ensures that the change data is available for downstream processing and auditing. The VACUUM command respects this retention period and does not delete files that are still within the retention window.
Verify the retention period for Change Data Feed and ensure that the VACUUM command's retention period is greater than or equal to the CDF retention period.
-- Check the current retention period for Change Data Feed
DESCRIBE HISTORY my_delta_table;
-- Adjust the retention period for the VACUUM command
VACUUM my_delta_table RETAIN 168 HOURS;
if you want to change the default retention period of change data feed then do this :
ALTER TABLE my_delta_table
SET TBLPROPERTIES ('delta.changeDataFeed.retentionDuration' = '30 days');
ALTER TABLE my_delta_table
SET TBLPROPERTIES ('delta.changeDataFeed.retentionDuration' = '30 days');
