topic Re: Discover and redact pii in Data Engineering

Discover and redact pii

Algocrat — Tue, 23 Jul 2024 15:42:56 GMT

Hi! What is the best way to discover and redact pii. Does Databricks offer any frameworks, or set of methods, or processes that we may follow?

Re: Discover and redact pii

szymon_dybczak — Tue, 23 Jul 2024 19:30:20 GMT

Hi @Algocrat ,

I think there is no out of the box solution in datbricks yet for this problem.

You can try to use some open source tools like Presidio ( look at below article).

Or if you are on azure, you can try to use Azure Purview

https://medium.com/databricks-platform-sme/identifying-and-tagging-pii-data-with-unity-catalog-870522f25730

Re: Discover and redact pii

viswesh — Fri, 03 Jan 2025 02:14:27 GMT

Hey @Algocrat @szymon_dybczak , just wanted to let you know that Databricks is currently working on a product to tackle PII / sensitive data classification. If you're a current customer, we recommend you reach out to your account representative to learn more about how to try it out!