Delta Lake does automatically clean up _delta_log files (JSON, CHECKPOINT, CRC), but only when two conditions are met:

1. The retention durations are respected
   By default:

   • delta.logRetentionDuration = 30 days

   • delta.deletedFileRetentionDuration = 7 days

   • spark.databricks.delta.retentionDurationCheck.enabled = true (safety check)

2. A new checkpoint is created after the retention window has passed
   Cleanup only happens when a new checkpoint is written, not immediately when properties are changed.

✔️ Why files are not being deleted in your case

Even though you set:

SET spark.databricks.delta.retentionDurationCheck.enabled = false;

ALTER TABLE table_name
SET TBLPROPERTIES (
  'delta.logRetentionDuration'='interval 1 minutes',
  'delta.deletedFileRetentionDuration'='interval 1 minutes'
);

VACUUM table_name RETAIN 0 HOURS;

Delta still won't delete older log files unless:

• The retention interval has actually passed in wall-clock time

• A new checkpoint is written after the retention window

• The table has enough new commits to trigger a checkpoint (usually every 10 commits)

Simply setting the retention to 1 minute does not retroactively delete anything. Delta only evaluates retention at checkpoint-creation time.

1. VACUUM does not delete JSON / CHECKPOINT log files

VACUUM only removes data files that are no longer referenced.
It never touches the transaction log.

This is why your _delta_log folder still looks large.

2. _delta_log cleanup only happens during checkpoint creation

If you are not generating new transactions, no cleanup will happen.

3. Very low retention settings (like 1 minute) are not recommended

They can cause checkpoint conflicts and metadata corruption during concurrent writes.

You can force a cleanup safely

1. Make sure retention check is disabled within the cluster that writes the table

SET spark.databricks.delta.retentionDurationCheck.enabled = false;

2. Set realistic, low—but safe—retention, e.g.:

ALTER TABLE table_name
SET TBLPROPERTIES (
  'delta.logRetentionDuration'='interval 1 day',
  'delta.deletedFileRetentionDuration'='interval 1 day'
);

3. Generate a few commits to trigger a new checkpoint:

df = spark.table("table_name").limit(1)
df.write.mode("append").format("delta").saveAsTable("table_name")

Repeat 10 times to force a checkpoint.

4. After the new checkpoint, older log files (beyond retention) will be removed automatically.

In Summary:

• _delta_log files are not deleted by VACUUM

• They are only deleted during checkpoint creation

• Changing retention properties does not delete old logs immediately

• You must generate commits and allow a checkpoint to be created

• Only then will Delta remove logs older than the retention window