- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-30-2025 08:00 AM
logRetentionDuration: "interval 1 days"
I was doing some table operations and see in the _delta_log folder files such as
00000000000000000000.json 00000000000000000001.json 00000000000000000002.json
After one day, I was doing some other operations and I see that all 10 commit compacted files are created in the log. However, I was expecting that the old log jsons (older than 1 day) are removed. Why are they still there?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-01-2025 06:09 AM
Hello @a_user12 ,
deltaLogRetentionDuration is the interval after which the delta log files will be removed from the delta log. Delta Lake adheres to a set of internal rules to clean up the delta log when the retention duration is exceeded.
Setting delta.logRetentionDuration alone will not automatically remove delta log files. Databricks uses a specific internal cleanup logic, and log files are only deleted once both retention criteria and the required checkpoint files are present. In other words, simply configuring the retention duration does not directly trigger the removal of delta log files: Delta Lake manages delta log deletion asynchronously, and deletes only files eligible per retention rules after periodic checkpointing
