Vector index not syncing: DELTA_UNSUPPORTED_TIME_TRAVEL_BEYOND_DELETED_FILE_RETENTION_DURATION

[FantineM]

Hi All,

Lately I have had issues with my vector search index not syncing.
The associated pipeline fails to create with error:

failed to resolve flow: '__online_index_view'. 
com.databricks.sql.transaction.tahoe.DeltaAnalysisException: [DELTA_UNSUPPORTED_TIME_TRAVEL_BEYOND_DELETED_FILE_RETENTION_DURATION] Cannot time travel beyond delta.deletedFileRetentionDuration (168 HOURS) set on the table.

I am guessing this means the source table for the index? This happened on an existing index which had never posed any issue. I deleted and recreated the index (7 days ago!) and now it is happening again.
If anyone has experienced the same issue and resolved, I'd appreciate any help or advice!

Thanks in advance!

[szymon_dybczak]

Hi @FantineM ,

Check if that table has predictive optimization enabled. If that's the case than probably that's a root cause. 
Predictive optimization will run VACCUM automatically.

Predictive optimization for Unity Catalog managed tables - Azure Databricks | Microsoft Learn

DESCRIBE TABLE EXTENDED your_table_name

To avoid this error, you can just disable predictive optimization. How to do that you can find here:

Predictive optimization for Unity Catalog managed tables - Azure Databricks | Microsoft Learn

If my answer was helpful, please consider marking it as accepted solution

[FantineM]

It seems like it could be it but I do have a few more questions if that's ok:
- is this a new thing or has it been enabled automatically recently?
- what would be the consequences if I disabled that (I understand it would be at the schema a minima) for other tables, considering this is a production environment?
- if I do disable it for this schema, will it fix the sync pipeline or is that data lost forever now (meaning I still have to recreate the index)
- and last, this is quite a big caveat for index source tables, should it be reported and to whom?

Thanks a million for your kind and very fast answer!

[szymon_dybczak · Accepted solution]

Hi, 

1. As docs says: "Predictive optimization is enabled by default for accounts created on or after November 11, 2024. Databricks began enabling existing accounts on May 7, 2025. This rollout is gradual and is expected to complete by April 2026. "

So I guess in your case that feature was enabled recently.

2. The only consequence is that VACCUM/ANALYZE/OPTIMIZE won't be performed automatically. But you can disable it for a single table. I'm quite surprised that they didn't mentioned that in docs:

%sql
ALTER TABLE your_table  DISABLE PREDICTIVE OPTIMIZATION

3. No, basically VACCUM operation removed those files. In Azure you can try to restore files that were vaccumed if you have enabled soft delete option on storage account. 

4. TBH I don't know. You can create support ticket I guess?

[FantineM]

Thanks again for the kind help!