Are Delta tables able to support GDPR compliance?

User16826992666 · ‎06-16-2021

I know that when deletes are made from a Delta table the underlying files are not actually removed. For compliance reasons I need to able to truly delete the records. How can I know which files need to be removed, and is there a way to remove them other than manually?

sajith_appukutt · ‎06-18-2021

Here is a document explaining best practices for GDPR and CCPA compliance using Delta Lake.

Specifically on cleaning up stale data - you can use the VACUUM function to remove files that are no longer referenced by a Delta table and are older than a specified retention threshold - preventing the ability to view history and roll back.

For e.g.

VACUUM gdpr.customers RETAIN 100 HOURS

View solution in original post