- 7535 Views
- 9 replies
- 10 kudos
Clean up _delta_log files
Hello experts. We are trying to clarify how to clean up the large amount of files that are being accumulated in the _delta_log folder (json, crc and checkpoint files). We went through the related posts in the forum and followed the below:SET spark.da...
- 7535 Views
- 9 replies
- 10 kudos
- 10 kudos
Delta Lake does automatically clean up _delta_log files (JSON, CHECKPOINT, CRC), but only when two conditions are met: The retention durations are respectedBy default: delta.logRetentionDuration = 30 days delta.deletedFileRetentionDuration = 7 day...
- 10 kudos
- 11045 Views
- 7 replies
- 0 kudos
Log files are not getting deleted automatically after logRetentionDuration internal
Hi team Log files are not getting deleted automatically after logRetentionDuration internal from delta log folder and after analysis , I see checkpoint files are not getting created after 10 commits . Below table properties using spark.sql( f""" ...
- 11045 Views
- 7 replies
- 0 kudos