I have noticed what seems to be unexpected behavior with the history of Unity Catalog managed tables and would like to understand whether this is expected.
As a test, I created a table with two versions:
- Version 0
- Version 1 (created approximately 200 hours ago)
Version 1 is still the current version of the table. At this point, the following query works as expected:
SELECT * FROM table VERSION AS OF 1;
Since version 1 is the current version, the query returns the current contents of the table.
Next, I make a small change to the table (for example, deleting a single row), which creates version 2. Version 1 is now a historical version rather than the current version.
Immediately after version 2 is created, running the exact same query:
SELECT * FROM table VERSION AS OF 1;
returns the following error:
Cannot time travel beyond delta.deletedFileRetentionDuration (168 HOURS) set on the table.
To my knowledge, no manual VACUUM operation was executed. The error occurs immediately after the new version is created.
What I find surprising is that version 1 was fully accessible while it was the current version, but as soon as a newer version is created, it becomes inaccessible because it is older than the 168-hour retention period.
My understanding was that delta.deletedFileRetentionDuration controls how long deleted files are retained, and that historical versions should remain accessible unless the required files have been physically removed (for example, by a VACUUM operation).
Is this behavior expected for Unity Catalog managed tables? Are there additional settings, such as Predictive Optimization or automatic maintenance processes, that could cause this behavior even when no explicit VACUUM command has been run?
Extra information: Predictive Optimization is enabled in this catalog, but to my knowledge it doesn't run immediately after any query.
Any clarification would be appreciated.
