What you’re seeing is expected behavior — the _delta_log folder always keeps a history of JSON commit files, checkpoint files, and CRCs. Even if you lower delta.logRetentionDuration and run VACUUM, cleanup won’t happen immediately. A couple of points to note:

• The property delta.logRetentionDuration controls how long log history is kept for time travel, but actual cleanup only happens when a new checkpoint is written and retention thresholds are met.

• Setting it to something like 1 minute will disable time travel almost immediately, but you still need to wait for the next compaction/checkpoint cycle to actually drop files.

• VACUUM only removes data files, not log files — so it won’t reduce _delta_log size on its own.

If you really don’t need any history/time travel, the supported approach is to:

1. Set spark.databricks.delta.retentionDurationCheck.enabled = false.

2. Use a very small delta.logRetentionDuration (like interval 1 minute).

3. Trigger a few commits (inserts/updates) so new checkpoints are written.

4. Delta will then automatically prune older JSON and CRC files beyond the retention window.

Also note that the _delta_log folder will never be completely empty — at least the most recent checkpoint plus a few commit files are always retained.