Table history time travel

MVMZ — Tue, 16 Jun 2026 12:12:32 GMT

I have noticed what seems to be unexpected behavior with the history of Unity Catalog managed tables and would like to understand whether this is expected.

As a test, I created a table with two versions:

Version 0
Version 1 (created approximately 200 hours ago)

Version 1 is still the current version of the table. At this point, the following query works as expected:

SELECT * FROM table VERSION AS OF 1;

Since version 1 is the current version, the query returns the current contents of the table.

Next, I make a small change to the table (for example, deleting a single row), which creates version 2. Version 1 is now a historical version rather than the current version.

Immediately after version 2 is created, running the exact same query:

SELECT * FROM table VERSION AS OF 1;

returns the following error:

Cannot time travel beyond delta.deletedFileRetentionDuration (168 HOURS) set on the table.

To my knowledge, no manual VACUUM operation was executed. The error occurs immediately after the new version is created.

What I find surprising is that version 1 was fully accessible while it was the current version, but as soon as a newer version is created, it becomes inaccessible because it is older than the 168-hour retention period.

My understanding was that delta.deletedFileRetentionDuration controls how long deleted files are retained, and that historical versions should remain accessible unless the required files have been physically removed (for example, by a VACUUM operation).

Is this behavior expected for Unity Catalog managed tables? Are there additional settings, such as Predictive Optimization or automatic maintenance processes, that could cause this behavior even when no explicit VACUUM command has been run?

Extra information: Predictive Optimization is enabled in this catalog, but to my knowledge it doesn't run immediately after any query.

Any clarification would be appreciated.

Re: Table history time travel

Ashwin_DSA — Tue, 16 Jun 2026 21:00:48 GMT

Hi @MVMZ,

What you’re seeing is expected for Unity Catalog managed tables.

The key detail is that for Unity Catalog managed tables, Databricks blocks time travel queries when the requested version is older than delta.deletedFileRetentionDuration, which is 7 days by default. This is called out in the public documentation for table history and time travel.

That explains why SELECT * FROM table VERSION AS OF 1 works while version 1 is still the current version, but starts failing as soon as version 2 is created. Before version 2 exists, you are effectively reading the current table state. Once version 2 is committed, version 1 becomes a historical version, and because it is already older than 168 hours, the query is treated as a time travel request beyond the allowed retention window and is blocked immediately.

So in this case, the behaviour is not dependent on whether you manually ran VACUUM. The current behaviour for Unity Catalog managed tables is that the query can be blocked based on the table’s delta.deletedFileRetentionDuration setting, even if the older files have not yet been manually cleaned up by you. The docs also explain that to query an older version successfully, Databricks needs both the log and the underlying data files for that version to still be retained, and that time travel should generally be planned around the VACUUM retention threshold.

Predictive Optimization can also be relevant here, because for Unity Catalog managed tables it can automatically run VACUUM, OPTIMIZE, and ANALYZE in the background. So even if nobody explicitly issued a VACUUM command, old files can still be cleaned up automatically when Predictive Optimization is enabled.

If the requirement is to keep older versions queryable for longer than 7 days, the fix is to increase delta.deletedFileRetentionDuration before relying on that history. The data retention section of the docs also notes that if you want, for example, 30 days of historical access, you should increase delta.deletedFileRetentionDuration accordingly, and ensure log retention is at least as long as well.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

topic Table history time travel in Data Engineering

Table history time travel

Re: Table history time travel