Unexpected behavior of Delta VACUUM – need explanation

[bricks_2026]

Hello

I have a question about how VACUUM works.

We executed the following VACUUM command from a PySpark job:
spark.sql("""VACUUM catalog_name.schema_name.table_name RETAIN 168 HOURS""")

The Delta table has a table property retention period of 365 days, but in our PySpark code we explicitly set:
spark.conf.set("spark.databricks.delta.retentionDurationCheck.enabled", "false")

The table history shows the following entries for VACUUM:
version timestamp operation operationParameters operationMetrics
67 2026-08-07 17:08:30 VACUUM END {"status":"COMPLETED"} {"numDeletedFiles":"0","numVacuumedDirectories":"54"}
66 2026-08-07 17:08:29 VACUUM START {"retentionCheckEnabled":"false","defaultRetentionMillis":"31536000000"} {"numFilesToDelete":"16","sizeOfDataToDelete":"0"}

1. In the VACUUM START entry, operationParameters shows "retentionCheckEnabled":"false" and "defaultRetentionMillis":"31536000000" (365 days), but it does not show the explicitly specified retention (168 hours).
Is this a known issue/behavior?
Genie suggests that when VACUUM is called programmatically via PySpark (for example, spark.sql("VACUUM ... RETAIN 168 HOURS")) on certain Databricks Runtime versions, the specifiedRetentionMillis field is sometimes not logged in the table history, whereas it appears when the same command is run interactively in SQL. Can anyone confirm whether this is a known limitation or bug, and if it depends on specific Databricks Runtime versions?

2. Why were no files deleted?
We expected that, after running VACUUM ... RETAIN 168 HOURS, older versions would be removed so that time travel to those versions is no longer possible. Specifically, we have a key that was updated several months ago. After the VACUUM operation, we can still perform time travel to an older version of that key.

In the history:
operationMetrics for VACUUM START shows "numFilesToDelete":"16".
operationMetrics for VACUUM END shows "numDeletedFiles":"0".
Why were 0 files deleted, even though 16 files were reported as candidates? Under which conditions would VACUUM decide not to delete any of the candidate files, even after we disabled the retention duration check and used RETAIN 168 HOURS?

Any explanation of this behavior or pointers to relevant documentation would be very helpful.

[Louis_Frolio]

Greetings @bricks_2026 , I did some digging and here is what I found.

First, good question, and you've already pulled the right evidence from the table history. This looks like retention semantics rather than a PySpark quoting issue or a logging bug. Let me walk through both questions.

Question 1: why doesn't the history show the 168-hour retention?

When a RETAIN clause actually drives a VACUUM run, the START entry records it as specifiedRetentionMillis next to retentionCheckEnabled and defaultRetentionMillis. You can see an example in this thread: https://community.databricks.com/t5/data-engineering/vacuum-command-runs-without-any-retention-perio...

Your START entry has no specifiedRetentionMillis and shows defaultRetentionMillis: 31536000000. That tells us the run used your table's 365-day retention property, not 168 hours. And on current runtimes, that's documented behavior, not a defect. The retention window for VACUUM is now controlled by the delta.deletedFileRetentionDuration table property. This applies in Databricks Runtime 18.0 and above, and for Unity Catalog managed tables it applies as far back as Runtime 13.3 LTS. Since you're using a three-level catalog.schema.table name, that likely covers you. Notice the current VACUUM syntax doesn't even document a RETAIN clause anymore: https://docs.databricks.com/aws/en/sql/language-manual/delta-vacuum and https://docs.databricks.com/aws/en/tables/operations/vacuum

So the answer to "is this a known issue or behavior" is: known behavior, assuming your runtime and table type match the above. Disabling spark.databricks.delta.retentionDurationCheck.enabled removes the safety check, but it doesn't make RETAIN override the table property. I'd confirm your exact runtime and whether the table is managed or external, since that determines which rule applies. One heads up if you ever run this on serverless: that Spark config isn't supported there at all: https://kb.databricks.com/delta/the-deltaretentiondurationcheck-property-is-not-recognized-when-usin...

Question 2: why can you still time travel to the old key?

Given the above, the effective retention for your run was 365 days. A key updated several months ago is well inside that window, so its old files were never eligible for deletion, and time travel to those versions still works. Fully consistent, if not what you intended. Reference on retention and time travel: https://docs.databricks.com/aws/en/tables/history

One more thing to check, because it bites people even after retention is fixed: deletion vectors. If delta.enableDeletionVectors is true, an UPDATE marks old rows as deleted in metadata instead of rewriting the Parquet file. The original file stays referenced by the current version, and VACUUM never deletes files the current version references. To physically remove those old values, run REORG TABLE ... APPLY (PURGE) first, then VACUUM after the retention window passes: https://docs.databricks.com/aws/en/sql/language-manual/delta-reorg-table

The 16 candidates versus 0 deleted

Those metrics describe different stages. numFilesToDelete counts paths identified during the scan; numDeletedFiles counts paths where the actual delete call succeeded. Your sizeOfDataToDelete: 0 says the 16 candidates were zero-byte objects, so they weren't your key's data files in any case; they're likely non-data paths or directory markers, and your END entry's numVacuumedDirectories: 54 shows directory cleanup is tracked separately. In the open-source implementation, a path that has already disappeared or whose delete returns false counts as a candidate but not a deletion: https://github.com/delta-io/delta/blob/master/spark/src/main/scala/org/apache/spark/sql/delta/comman...

I'll be candid that I can't pin down those 16 paths from the history alone. A DRY RUN will show you exactly what's eligible, and the KB article on reading these metrics is here: https://kb.databricks.com/delta/track-deleted-files-from-vacuum-in-delta-table-history

What I'd do next

1. Confirm the runtime version of the job and whether the table is a Unity Catalog managed table (DESCRIBE DETAIL shows this).
2. Run SHOW TBLPROPERTIES catalog_name.schema_name.table_name and check delta.deletedFileRetentionDuration, delta.enableDeletionVectors, and delta.logRetentionDuration.
3. If seven days is the intended window, set it deliberately: ALTER TABLE ... SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = 'interval 7 days').
4. Run VACUUM catalog_name.schema_name.table_name DRY RUN to inspect the candidate paths, then VACUUM ... FULL without relying on RETAIN.
5. If deletion vectors are enabled and you need old values physically gone, run REORG TABLE ... APPLY (PURGE) first, then VACUUM once the retention window allows.
6. If a DRY RUN lists real data files but VACUUM still reports zero deletions, compare those paths against cloud storage and check the compute identity's delete permissions. At that point I'd open a support ticket with both history entries and the DRY RUN output attached.

One caution before shrinking retention: dropping from 365 days to 7 days permanently removes time travel beyond a week once VACUUM runs, and on newer runtimes time travel requests older than the property are blocked outright. Verify no readers, streaming jobs, or recovery requirements need the longer history.

The takeaway: your VACUUM did exactly what the table property told it to do. Set delta.deletedFileRetentionDuration to your intended window, check for deletion vectors, and the behavior should match your expectations.

Regards, Louis

[bricks_2026]

Hello Louis,

Thank you so much for the incredibly detailed and helpful answer – it was a great help. I have the following question:

We have a daily job that loads data from one Databricks schema into one or more Databricks schemas using CDF.
In addition, we have a weekly job that runs VACUUM on the tables in the source schema.

The VACUUM logic identifies downstream consumers via Unity Catalog lineage and determines how far behind each target is.
It always retains at least seven days of source CDF history, plus an additional 24‑hour buffer for older or lagging consumers, and then uses the most conservative (largest) value across all consumers.
The retention period for each table is calculated based on this logic.

In simple terms: we do not want to vacuum data that has not yet been read by downstream consumers (target schemas).

Based on the calculated retention period, we first set the source schema tables’ deletedFileRetentionDuration and delta.logRetentionDuration to that value, and then execute VACUUM.
So far, this works as expected.

The problem is that the window is moving.
For example, on 13 August we calculate the table’s retention period (say, 7 days) and set deletedFileRetentionDuration to 7 days.
However, because this is a rolling window, each day we effectively lose 24 hours from that original 7‑day buffer.

Four days after VACUUM runs, the table property still shows deletedFileRetentionDuration = 7 days, but the actual effective window has moved forward.
This happens because deletedFileRetentionDuration is a relative value, and our VACUUM job only runs weekly resulting in moving or sliding window.
As mentioned above, the calculation of the retention period, the update of deletedFileRetentionDuration, and the VACUUM execution all occur only once a week.

I don’t think this is a problem unique to us; it seems like a common pattern.

What are your thoughts on this?
Are there any recommendations from Databricks on how to handle this ?

[Louis_Frolio]

---

Greetings @bricks_2026 , glad the first answer helped. Your sliding window observation is sharp, but one correction to the mental model: delta.deletedFileRetentionDuration only drives physical deletion at the moment a VACUUM executes. Between your weekly runs, nothing gets removed just because the calendar advances. The window re-anchors to "now" each time VACUUM runs.

That said, the stale property still bites in three ways:

1. Predictive optimization. On Unity Catalog managed tables with predictive optimization enabled, Databricks can trigger VACUUM automatically between your weekly runs, using the current property anchored to that moment. Check this first: https://docs.databricks.com/aws/en/sql/language-manual/delta-vacuum
2. Read gating. On newer runtimes, time travel and CDF reads beyond the property can be blocked even when the files still exist, so a lagging consumer can fail mid-week.
3. Log cleanup. delta.logRetentionDuration is enforced automatically at checkpoint time on your daily writes, not by your weekly job. Tie it to the same tight value and commit history ages out daily, and CDF needs those commits regardless of whether the data files survive: https://docs.databricks.com/aws/en/tables/operations/vacuum

What I'd recommend:

1. Pad the retention by your recompute interval: max consumer lag + your 24 hour buffer + the job cadence (7 days). This alone fixes the drift.
2. Better: recompute consumer lag and update deletedFileRetentionDuration daily, keep VACUUM weekly. Then your 24 hour buffer is enough. VACUUM ... LITE makes frequent runs cheap on large tables.
3. Keep delta.logRetentionDuration static and generous (30 day default or higher). Log JSON is cheap, and losing commits breaks CDF reads.
4. For long-lagging consumers, persist CDF output to a durable audit table so source retention never has to cover them.

I'm not aware of official guidance for lineage driven sliding retention like yours; items 1 and 2 are standard practice, not a documented recommendation.

The takeaway: the property only has teeth when something enforces it. Recompute it as often as it can be enforced, or pad it to cover the gap.

If you find this helpful please Accept as  Solution so that others can have confidence in the answer.

Cheers, Lou.