Hi

The error DELTA_UNSUPPORTED_TIME_TRAVEL_BEYOND_DELETED_FILE_RETENTION_DURATION confirms that the underlying files required for Version 25 have been deleted from the storage. Since the metadata knows those files should be there but finds them gone, it blocks the query.

1. What Databricks feature/job is triggering this automatic VACUUM on managed tables?

The service principal in the logs is Databricks Service executing Predictive Optimization automatically. Predictive Optimization is the standard for Unity Catalog managed tables that automatically handles OPTIMIZE and VACUUM operations in the background using serverless compute. It targets tables where it detects high file fragmentation or a build up of expired snapshots to maintain performance and to reduce storage costs.

2. How can I override/disable this for specific tables (e.g. increase retention or opt out)?

You can 

• Increase Retention - Delta keeps 7 days (168 hours) of history. You can increase it using below

  ALTER TABLE eud_poland.staging.pibb_extract_preprocessed 
  SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = '30 days');

• Opt out - You can disable the service for a specific catalog or schema. More details here

  You must manually manage the tables for optimizations to avoid performance degradation if you disable it.

3. is recovery of that old version only possible (its managed table)?

No. Once VACUUM is complete and the files are deleted from the storage, the old state of the data is gone.

4. How should I back up a managed Delta table so I can recover older versions even after VACUUM (e.g. copy table)?

Time Travel is not a long-term backup solution. You can use Delta Deep Clone for long term backup

• Deep Clone - You can create a separate physical copy of the data and metadata.

  CREATE TABLE eud_poland.staging.pibb_extract_backup
  DEEP CLONE eud_poland.staging.pibb_extract_preprocessed;

5. Why is only this table affected?

Predictive Optimization does not hit every table with the same frequency. This specific table is targeted because:

1. You are doing frequent operations on this table creating many files that trigger the optimization threshold.

2. Table Size/Growth: The service prioritizes tables where storage savings or performance gains are most significant.