GDPR/CCPA Compliance Delete for PII data

[abhijit007]

Hi,

I am currently designing a PII governance framework to meet CCPA compliance requirements on Databricks. I understand that Databricks provides mechanisms such as VACUUM and Deletion Vectors combined with REORG … APPLY (PURGE) to permanently remove data. With a well‑designed deletion workflow across the Lakeflow / medallion architecture, end‑to‑end PII deletion can be achieved.

However, I would like to understand whether Databricks offers any native feature, service, or managed capability that can reduce the operational overhead of implementing and maintaining this workflow, and help centrally orchestrate and enforce PII deletions across the entire lakehouse, rather than relying primarily on custom pipelines and control tables.

Thanks in advance and really appreciate your response. 

[Sumit_7]

Hi @abhijit007,

No, Databricks still does NOT provide a native, centralized PII deletion orchestration service across the lakehouse. Though as you mentioned right, it's achievable through custom pipelines and control tables.
Check this - Prepare your data for GDPR compliance | Databricks on AWS

Thanks.

[abhijit007]

Hi @Sumit_7 ,

Thanks for the details. It's helpful.

[aleksandra_ch · Accepted solution]

Hi @abhijit007 ,

A new Data Classification feature (currently in Public Preview), allows to automatically classify and tag sensitive data in your catalog. It goes through few steps:

1. AI-driven engine scans Unity Catalog tables and detects PII data and assigns classification tags;
2. Results of classification are stored in a system table system.data_classification.results;
3. You can leverage ABAC policies using those tags to mask/filter PII;
4. Leverage the system table to automatically remove GDPR data.

Check for more details:

• Data classification documentation
• GDPR discovery and deletion notebook example

Best regards,

[abhijit007]

Hi @aleksandra_ch ,

Thanks .. The notebook reference is helpful.

[Armanillo_RoK]

Hi there,
   I spent the last while building a metadata-driven GDPR retention and right-to-erasure framework on Databricks Unity Catalog.

The core decision: policy lives in a small set of governance tables (what's PII, retention rules, active legal holds, erasure requests), and a job engine just walks them. Adding a new table to the framework means inserting registry rows, not writing new code.

For base retention we adopted the new Auto-TTL feature, layered with a hold-aware override so litigation holds or open erasure requests can gate pure age-based expiry. RTBF requests hit redaction before physical purge, so we have a defensible "stopped processing this data on date X" answer well before storage cleanup runs.

Structure inspired by Snowflake-Labs' sfguide-data-retention-and-purge (Apache 2.0), adapted for Unity Catalog with automatic PII discovery and Auto-TTL added.

Still open: propagating an erasure request across systems where the same person shows up under different source-system keys. The framework assumes one entity, one key today. If you've solved that cleanly, I'd like to hear how.

Happy GDPRing!

[WilliamCarterr]

For GDPR/CCPA deletion requests involving PII, I think the key is having a clearly defined process that identifies all locations where the user’s data may exist, including primary records, backups, logs, integrations, and derived data.

It’s also important to distinguish between data that must be deleted and information that may need to be retained for a legitimate legal or operational requirement. Keeping an audit trail of the deletion request without retaining unnecessary PII can help demonstrate that the request was handled appropriately.

Automating the workflow where possible could reduce mistakes, but I’d still include validation and access controls before permanently deleting data. Clear retention policies and documented deletion procedures can make compliance much easier to manage.

For a quick break after working through privacy requirements, learntofly2.io can be a simple browser option.