File optimization for delta table (versioning and snapshot ) in storage S3
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ02-13-2023 11:14 PM
Delta table generates new file for every insert or update on table and keep the old version files also for versioning and time travel history .
I have 1tb data as delta table and every 30 minutes , 90 percent data getting updated so file size will be getting increase exponentially .
Is there any way to optimize the storage size ( want to keep 1 week time travel history also)
- Labels:
-
Delta
-
Delta table
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ02-14-2023 12:38 AM
Hi @vinay kumarโ ,
You can use partitionBy() to partition your data that's help you for data skipping and use vacuum to delete older history from delta tables.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ02-14-2023 01:09 AM
Yes , but partitionBy will help at some level to skip new file creation - but lets assume 90 percent data getting updated on every load and want to keep the time travel history also.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ02-21-2023 02:21 AM
Hi @vinay kumarโ
Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.
We'd love to hear from you.
Thanks!