Anonymous
Not applicable
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-04-2021 10:29 AM
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-07-2021 07:40 AM
It really depends on your business intentions! You can remove files no longer referenced by a Delta table and are older than the retention threshold by running the vacuum command on the table. vacuum is not triggered automatically. The default retention threshold for the files is 7 days.
The main reason I recommend VACUUM-ing is for compliance and to reduce storage costs. If you need to remove older copies of the date for compliance, I would VACUUM as quickly as processing is complete to ensure the records are removed. To manage storage costs, you will need to balance the cost of keeping older copies of data with the storage cost for them, and that is dependent on each use case.
The main reason I recommend VACUUM-ing is for compliance and to reduce storage costs. If you need to remove older copies of the date for compliance, I would VACUUM as quickly as processing is complete to ensure the records are removed. To manage storage costs, you will need to balance the cost of keeping older copies of data with the storage cost for them, and that is dependent on each use case.