The functionality of table property delta.logRetentionDuration
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-27-2022 09:01 PM
We have one project requirement where we have to store only the 14 days history for delta tables. So for testing, I have set the delta.logRetentionDuration = 2 days using the below command
spark.sql("alter table delta.`[delta_file_path]` set TBLPROPERTIES (’delta.logRetentionDuration'='interval 2 days’)")
However, I tried it after specific intervals, i.e., (after two days) I can still time travel back to previous versions. Do we need to run Vacuum after setting this property, or it works only for >30 days?
Can I please get help on this?
Also, will it physically delete the data files or will only log files be deleted?
Labels:
- Labels:
-
Delta Tables
-
Table