Delta Tables: Time-To-Live

[Michael_Appiah]

I have seen somewhere (might have been in a Databricks Tech Talk) a Delta Table feature which allows to specify the "expiration date" of data stored in Delta Tables. Once rows surpass their time-to-live, they are automatically deleted or archived. 

Does anyone know if such a feature is already out there or if it is currently in development?

Unfortunately, I cannot find the tech talk, doc or blog post where I saw this but I am sure I saw it somewhere :D...

[Michael_Appiah]

Hi Diana, 

thank you for your answer. 

However,  the delta.logRetentionDuration configuration only specifies how long the Delta Table history should be retained (https://docs.delta.io/latest/table-properties.html#:~:text=delta.logRetentionDuration). What I was looking for was a feature which allows users to specify in the Delta Table itself (for example in a "time_to_live" column) the expiration date of a given data set. Once that data set surpasses that expiration date, the row will be deleted from the Delta table (deletion might be triggered by another operation like "optimize" similarly to how deletion vectors work).

I am sure I saw a Databricks announcement regarding such a feature but unfortunately cannot find it anymore 😢
But yeah, Im not ruling out that I am mistaken or that this is just wishful thinking on my end 😂

[christian_ligus]

You are not hallucinating, I remember the talk about this feature too, and so does the AI assistant! Copy-pasted directly from it, although it seems the feature was never implemented:

Using Time-To-Live (TTL)

One approach to handle this is by using Time-To-Live (TTL) features provided by some databases and data storage systems. However, since you are using Databricks, you can utilize the Delta Lake feature which supports automatic data expiration.

<bunch of trash code where the methods don't exist>

So... any plans to implement this, Databricks? It'd be great to have this.

[emma_s]

Hi, you did not make this up, this feature is currently in Private preview, if you want to try out the private preview then speak to your account team.

[Hubert-Dudek]

Yes that feature was announced on Data + AI summit - really cool.