cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

A Surprise Findings in Delta Live Table

CURIOUS_DE
New Contributor III

While DLT has some powerful features, I found myself doing a double-take when I realized it doesnโ€™t natively support hard deletes. Instead, it leans on a delete flag identifier to manage these in the source table. A bit surprising for a tool of its caliber, donโ€™t you think? @Sujitha 

Let's pass it on to all Databricks Community User

1 REPLY 1

VZLA
Databricks Employee
Databricks Employee

Thanks for your feedback !

I believe, Delta Live Tables (DLT) does not natively support hard deletes and instead uses a delete flag identifier to manage deletions, a design choice rooted in ensuring compliance with regulations like GDPR and CCPA. This approach facilitates traceability, auditing, and the "Right to be Forgotten", avoiding the complexities and risks of immediate hard deletes in large-scale data environments. Additionally, it leverages Delta Lakeโ€™s time travel and retention features to maintain data integrity, support recovery, and prevent disruptions to downstream pipelines, striking a balance between governance, compliance, and operational reliability.

https://www.databricks.com/blog/handling-right-be-forgotten-gdpr-and-ccpa-using-delta-live-tables-dl...

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group