Unexpected behavior of Delta VACUUM – need explanation

bricks_2026 — Tue, 11 Aug 2026 10:02:23 GMT

Hello

I have a question about how VACUUM works.

We executed the following VACUUM command from a PySpark job:
spark.sql("""VACUUM catalog_name.schema_name.table_name RETAIN 168 HOURS""")

The Delta table has a table property retention period of 365 days, but in our PySpark code we explicitly set:
spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")

The table history shows the following entries for VACUUM:
version timestamp operation operationParameters operationMetrics
67 2026-08-07 17:08:30 VACUUM END {"status":"COMPLETED"} {"numDeletedFiles":"0","numVacuumedDirectories":"54"}
66 2026-08-07 17:08:29 VACUUM START {"retentionCheckEnabled":"false","defaultRetentionMillis":"31536000000"} {"numFilesToDelete":"16","sizeOfDataToDelete":"0"}

1. In the VACUUM START entry, operationParameters shows "retentionCheckEnabled":"false" and "defaultRetentionMillis":"31536000000" (365 days), but it does not show the explicitly specified retention (168 hours).
Is this a known issue/behavior?
Genie suggests that when VACUUM is called programmatically via PySpark (for example, spark.sql("VACUUM ... RETAIN 168 HOURS")) on certain Databricks Runtime versions, the specifiedRetentionMillis field is sometimes not logged in the table history, whereas it appears when the same command is run interactively in SQL. Can anyone confirm whether this is a known limitation or bug, and if it depends on specific Databricks Runtime versions?

2. Why were no files deleted?
We expected that, after running VACUUM ... RETAIN 168 HOURS, older versions would be removed so that time travel to those versions is no longer possible. Specifically, we have a key that was updated several months ago. After the VACUUM operation, we can still perform time travel to an older version of that key.

In the history:
operationMetrics for VACUUM START shows "numFilesToDelete":"16".
operationMetrics for VACUUM END shows "numDeletedFiles":"0".
Why were 0 files deleted, even though 16 files were reported as candidates? Under which conditions would VACUUM decide not to delete any of the candidate files, even after we disabled the retention duration check and used RETAIN 168 HOURS?

Any explanation of this behavior or pointers to relevant documentation would be very helpful.

Re: Unexpected behavior of Delta VACUUM – need explanation

Louis_Frolio — Wed, 12 Aug 2026 13:26:03 GMT

Greetings @bricks_2026 , I did some digging and here is what I found.

First, good question, and you've already pulled the right evidence from the table history. This looks like retention semantics rather than a PySpark quoting issue or a logging bug. Let me walk through both questions.

Question 1: why doesn't the history show the 168-hour retention?

When a RETAIN clause actually drives a VACUUM run, the START entry records it as specifiedRetentionMillis next to retentionCheckEnabled and defaultRetentionMillis. You can see an example in this thread: https://community.databricks.com/t5/data-engineering/vacuum-command-runs-without-any-retention-period-even-though-the/m-p/151616

Your START entry has no specifiedRetentionMillis and shows defaultRetentionMillis: 31536000000. That tells us the run used your table's 365-day retention property, not 168 hours. And on current runtimes, that's documented behavior, not a defect. The retention window for VACUUM is now controlled by the delta.deletedFileRetentionDuration table property. This applies in Databricks Runtime 18.0 and above, and for Unity Catalog managed tables it applies as far back as Runtime 13.3 LTS. Since you're using a three-level catalog.schema.table name, that likely covers you. Notice the current VACUUM syntax doesn't even document a RETAIN clause anymore: https://docs.databricks.com/aws/en/sql/language-manual/delta-vacuum and https://docs.databricks.com/aws/en/tables/operations/vacuum

So the answer to "is this a known issue or behavior" is: known behavior, assuming your runtime and table type match the above. Disabling spark.databricks.delta.retentionDurationCheck.enabled removes the safety check, but it doesn't make RETAIN override the table property. I'd confirm your exact runtime and whether the table is managed or external, since that determines which rule applies. One heads up if you ever run this on serverless: that Spark config isn't supported there at all: https://kb.databricks.com/delta/the-deltaretentiondurationcheck-property-is-not-recognized-when-using-serverless-compute

Question 2: why can you still time travel to the old key?

Given the above, the effective retention for your run was 365 days. A key updated several months ago is well inside that window, so its old files were never eligible for deletion, and time travel to those versions still works. Fully consistent, if not what you intended. Reference on retention and time travel: https://docs.databricks.com/aws/en/tables/history

One more thing to check, because it bites people even after retention is fixed: deletion vectors. If delta.enableDeletionVectors is true, an UPDATE marks old rows as deleted in metadata instead of rewriting the Parquet file. The original file stays referenced by the current version, and VACUUM never deletes files the current version references. To physically remove those old values, run REORG TABLE ... APPLY (PURGE) first, then VACUUM after the retention window passes: https://docs.databricks.com/aws/en/sql/language-manual/delta-reorg-table

The 16 candidates versus 0 deleted

Those metrics describe different stages. numFilesToDelete counts paths identified during the scan; numDeletedFiles counts paths where the actual delete call succeeded. Your sizeOfDataToDelete: 0 says the 16 candidates were zero-byte objects, so they weren't your key's data files in any case; they're likely non-data paths or directory markers, and your END entry's numVacuumedDirectories: 54 shows directory cleanup is tracked separately. In the open-source implementation, a path that has already disappeared or whose delete returns false counts as a candidate but not a deletion: https://github.com/delta-io/delta/blob/master/spark/src/main/scala/org/apache/spark/sql/delta/commands/VacuumCommand.scala

I'll be candid that I can't pin down those 16 paths from the history alone. A DRY RUN will show you exactly what's eligible, and the KB article on reading these metrics is here: https://kb.databricks.com/delta/track-deleted-files-from-vacuum-in-delta-table-history

What I'd do next

Confirm the runtime version of the job and whether the table is a Unity Catalog managed table (DESCRIBE DETAIL shows this).
Run SHOW TBLPROPERTIES catalog_name.schema_name.table_name and check delta.deletedFileRetentionDuration, delta.enableDeletionVectors, and delta.logRetentionDuration.
If seven days is the intended window, set it deliberately: ALTER TABLE ... SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = 'interval 7 days').
Run VACUUM catalog_name.schema_name.table_name DRY RUN to inspect the candidate paths, then VACUUM ... FULL without relying on RETAIN.
If deletion vectors are enabled and you need old values physically gone, run REORG TABLE ... APPLY (PURGE) first, then VACUUM once the retention window allows.
If a DRY RUN lists real data files but VACUUM still reports zero deletions, compare those paths against cloud storage and check the compute identity's delete permissions. At that point I'd open a support ticket with both history entries and the DRY RUN output attached.

One caution before shrinking retention: dropping from 365 days to 7 days permanently removes time travel beyond a week once VACUUM runs, and on newer runtimes time travel requests older than the property are blocked outright. Verify no readers, streaming jobs, or recovery requirements need the longer history.

The takeaway: your VACUUM did exactly what the table property told it to do. Set delta.deletedFileRetentionDuration to your intended window, check for deletion vectors, and the behavior should match your expectations.

Regards, Louis

topic Unexpected behavior of Delta VACUUM – need explanation in Data Engineering

Unexpected behavior of Delta VACUUM – need explanation

Re: Unexpected behavior of Delta VACUUM – need explanation