SET MANAGED AND ROLLBACK IN RELATION TO VACUUM

[Davide]

Hello everyone,

I am looking for clarification about the 14-day rollback period after converting an external table to a managed table using: ALTER TABLE table_name SET MANAGED.

Databricks documentation states that the conversion can be rolled back within 14 days. It also says that, after this period, Predictive Optimization or a manual VACUUM can delete data from the old external location to avoid unnecessary storage costs.
I would appreciate clarification on the following scenarios which are not documented (afaik):

1- Running VACUUM before day 14
If I run VACUUM before the 14-day rollback period ends, does it affect only files in the new managed-table location? Or can it also delete files from the original external location?

2- Retention period shorter than 14 days
If delta.deletedFileRetentionDuration is set to less than 14 days, for example 7 days: Does the rollback window become 7 days?
If I run VACUUM on day 8, could it delete data needed for rollback, or it will just affects the data in the managed location?

3- Retention period longer than 14 days
If delta.deletedFileRetentionDuration is set to more than 14 days, for example 20 days. Does the rollback window remain 14 days or it extends to 20 days?

My assumptions are that the rollback period is fixed to 14 days so, if you run vacuum before that period will not touch the data in the external location no matter the retention-days. If the retention is more than 20 days and if I run a vacuum or PO kicks in, the data in old location will be deleted.

Has anyone tested these cases or received clarification from Databricks? I could not find details about how the rollback period interacts with delta.deletedFileRetentionDuration, manual VACUUM, and Predictive Optimization.

Thanks for any guidance.

[DoTA]

Hi, I could not find these three cases documented either, so I will separate what the docs say from what I would treat as an assumption.

What is documented (Convert external tables to managed): Databricks keeps the data in the old external location for 14 days to allow rollback. After 14 days, if predictive optimization is enabled, it deletes that old data automatically. If predictive optimization is off, you run VACUUM on the converted managed table after the 14 days (needs DBR 17.3 LTS or above, or serverless). The doc also notes that even with predictive optimization on, the old data may not get deleted at day 14 for things like rarely used tables. Only the data files in the external location are removed. The Delta log and the Unity Catalog reference stay.

On your three questions:

1) VACUUM before day 14. The docs frame the cleanup of the old location as something that happens after the 14 days, and VACUUM is described as the manual route for that. They do not say what a VACUUM before day 14 does to the old location. I would not rely on "it only touches the new managed location". If rollback matters, simply do not run VACUUM (manually or through a job) on that table inside the window, and check whether predictive optimization is enabled on the schema.

2 and 3) delta.deletedFileRetentionDuration. The conversion page does not mention it at all. My reading is that this property controls which unreferenced files VACUUM may remove from the table's own storage, while the 14 day window is a separate fixed retention applied to the old external copy. That would mean a 7 day value does not shorten the rollback window and a 20 day value does not extend it. That is an inference, not a documented guarantee, so please do not treat my reading as confirmed.

Since you asked whether anyone has tested it, the safe way to settle it is a quick experiment on a small throwaway external Delta table: SET MANAGED, then run VACUUM with different retention settings and check whether the files in the old external path are still there (list the path before and after). Doing that on a test table costs minutes, and the result is better evidence than any of our assumptions. If you need it guaranteed for production, I would raise it with your Databricks account team so they can confirm in writing, since the docs leave this interaction unspecified.

Practical rule until then: after SET MANAGED, leave the table alone with respect to VACUUM and predictive optimization for the full 14 days if you want rollback to stay safe.

[Davide · Accepted solution]

Hello,

I was able to verify the behavior of the rollback period in relation to the table property retentionDays.

In the end, I confirmed that the rollback period is not affected by the retentionDays value. Regardless of whether retentionDays is set to a lower or higher value, the rollback period remains the same.

Therefore, changing the retentionDays setting does not impact the rollback window.