Delta Tables: Time-To-Live

[Michael_Appiah]

I have seen somewhere (might have been in a Databricks Tech Talk) a Delta Table feature which allows to specify the "expiration date" of data stored in Delta Tables. Once rows surpass their time-to-live, they are automatically deleted or archived. 

Does anyone know if such a feature is already out there or if it is currently in development?

Unfortunately, I cannot find the tech talk, doc or blog post where I saw this but I am sure I saw it somewhere :D...

[diana45peters]

---

@Michael_Appiah wrote:

I have seen somewhere (might have been in a Databricks Tech Talk) a Delta Table feature which allows to specify the "expiration date" of data stored in Delta Tables. Once rows surpass their time-to-live, they are automatically deleted or archived. 

Does anyone know if such a feature is already out there or if it is currently in development?

Unfortunately, I cannot find the tech talk, doc or blog post where I saw this but I am sure I saw it somewhere :D... USPS Liteblue

---

Yes, 

Databricks Delta Lake does have a feature called Time-To-Live (TTL) for data expiration. This feature allows you to specify an expiration date for data stored in Delta Tables, and once the data surpasses its TTL, it can be automatically deleted or archived.

You can configure the TTL settings using the delta.logRetentionDuration parameter to manage how long data is retained in the table. This can be particularly useful for managing data lifecycle and ensuring that old data is removed to free up storage space

Hope this helps!
Best regards,
Diana Peters

[Michael_Appiah]

Hi Diana, 

thank you for your answer. 

However,  the delta.logRetentionDuration configuration only specifies how long the Delta Table history should be retained (https://docs.delta.io/latest/table-properties.html#:~:text=delta.logRetentionDuration). What I was looking for was a feature which allows users to specify in the Delta Table itself (for example in a "time_to_live" column) the expiration date of a given data set. Once that data set surpasses that expiration date, the row will be deleted from the Delta table (deletion might be triggered by another operation like "optimize" similarly to how deletion vectors work).

I am sure I saw a Databricks announcement regarding such a feature but unfortunately cannot find it anymore 😢
But yeah, Im not ruling out that I am mistaken or that this is just wishful thinking on my end 😂