topic Delta Tables: Time-To-Live in Data Engineering

Delta Tables: Time-To-Live

Michael_Appiah — Wed, 27 Nov 2024 10:34:18 GMT

I have seen somewhere (might have been in a Databricks Tech Talk) a Delta Table feature which allows to specify the "expiration date" of data stored in Delta Tables. Once rows surpass their time-to-live, they are automatically deleted or archived.

Does anyone know if such a feature is already out there or if it is currently in development?

Unfortunately, I cannot find the tech talk, doc or blog post where I saw this but I am sure I saw it somewhere :D...

Re: Delta Tables: Time-To-Live

Michael_Appiah — Wed, 27 Nov 2024 11:48:19 GMT

Hi Diana,

thank you for your answer.

However, the delta.logRetentionDuration configuration only specifies how long the Delta Table history should be retained (https://docs.delta.io/latest/table-properties.html#:~:text=delta.logRetentionDuration). What I was looking for was a feature which allows users to specify in the Delta Table itself (for example in a "time_to_live" column) the expiration date of a given data set. Once that data set surpasses that expiration date, the row will be deleted from the Delta table (deletion might be triggered by another operation like "optimize" similarly to how deletion vectors work).

I am sure I saw a Databricks announcement regarding such a feature but unfortunately cannot find it anymore 😢
But yeah, Im not ruling out that I am mistaken or that this is just wishful thinking on my end 😂

Re: Delta Tables: Time-To-Live

christian_ligus — Wed, 10 Dec 2025 08:04:58 GMT

You are not hallucinating, I remember the talk about this feature too, and so does the AI assistant! Copy-pasted directly from it, although it seems the feature was never implemented:

Using Time-To-Live (TTL)

One approach to handle this is by using Time-To-Live (TTL) features provided by some databases and data storage systems. However, since you are using Databricks, you can utilize the Delta Lake feature which supports automatic data expiration.

So... any plans to implement this, Databricks? It'd be great to have this.

Re: Delta Tables: Time-To-Live

emma_s — Wed, 10 Dec 2025 09:08:51 GMT

Hi, you did not make this up, this feature is currently in Private preview, if you want to try out the private preview then speak to your account team.

Re: Delta Tables: Time-To-Live

Hubert-Dudek — Wed, 10 Dec 2025 09:37:20 GMT

Yes that feature was announced on Data + AI summit - really cool.