A Surprise Findings in Delta Live Table
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ08-27-2024 08:14 AM
While DLT has some powerful features, I found myself doing a double-take when I realized it doesnโt natively support hard deletes. Instead, it leans on a delete flag identifier to manage these in the source table. A bit surprising for a tool of its caliber, donโt you think? @Sujitha
Let's pass it on to all Databricks Community User
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ12-27-2024 10:13 AM
Thanks for your feedback !
I believe, Delta Live Tables (DLT) does not natively support hard deletes and instead uses a delete flag identifier to manage deletions, a design choice rooted in ensuring compliance with regulations like GDPR and CCPA. This approach facilitates traceability, auditing, and the "Right to be Forgotten", avoiding the complexities and risks of immediate hard deletes in large-scale data environments. Additionally, it leverages Delta Lakeโs time travel and retention features to maintain data integrity, support recovery, and prevent disruptions to downstream pipelines, striking a balance between governance, compliance, and operational reliability.

