A Surprise Findings in Delta Live Table

CURIOUS_DE — Tue, 27 Aug 2024 15:14:12 GMT

While DLT has some powerful features, I found myself doing a double-take when I realized it doesn’t natively support hard deletes. Instead, it leans on a delete flag identifier to manage these in the source table. A bit surprising for a tool of its caliber, don’t you think? @Sujitha

Let's pass it on to all Databricks Community User

Re: A Surprise Findings in Delta Live Table

VZLA — Fri, 27 Dec 2024 18:13:14 GMT

Thanks for your feedback !

I believe, Delta Live Tables (DLT) does not natively support hard deletes and instead uses a delete flag identifier to manage deletions, a design choice rooted in ensuring compliance with regulations like GDPR and CCPA. This approach facilitates traceability, auditing, and the "Right to be Forgotten", avoiding the complexities and risks of immediate hard deletes in large-scale data environments. Additionally, it leverages Delta Lake’s time travel and retention features to maintain data integrity, support recovery, and prevent disruptions to downstream pipelines, striking a balance between governance, compliance, and operational reliability.

https://www.databricks.com/blog/handling-right-be-forgotten-gdpr-and-ccpa-using-delta-live-tables-dlt

topic A Surprise Findings in Delta Live Table in Data Engineering

A Surprise Findings in Delta Live Table

Re: A Surprise Findings in Delta Live Table