Delta Tables: Time-To-Live

[Michael_Appiah]

I have seen somewhere (might have been in a Databricks Tech Talk) a Delta Table feature which allows to specify the "expiration date" of data stored in Delta Tables. Once rows surpass their time-to-live, they are automatically deleted or archived. 

Does anyone know if such a feature is already out there or if it is currently in development?

Unfortunately, I cannot find the tech talk, doc or blog post where I saw this but I am sure I saw it somewhere :D...

[Michael_Appiah]

Hi Diana, 

thank you for your answer. 

However,  the delta.logRetentionDuration configuration only specifies how long the Delta Table history should be retained (https://docs.delta.io/latest/table-properties.html#:~:text=delta.logRetentionDuration). What I was looking for was a feature which allows users to specify in the Delta Table itself (for example in a "time_to_live" column) the expiration date of a given data set. Once that data set surpasses that expiration date, the row will be deleted from the Delta table (deletion might be triggered by another operation like "optimize" similarly to how deletion vectors work).

I am sure I saw a Databricks announcement regarding such a feature but unfortunately cannot find it anymore 😢
But yeah, Im not ruling out that I am mistaken or that this is just wishful thinking on my end 😂