topic Re: Masking techniques for more PII columns in Get Started Discussions

Masking techniques for more PII columns

Phani1 — Thu, 01 Feb 2024 09:02:20 GMT

Hi Databricks Team,

We would appreciate it if you could inform us about the situations when Column-Masking, Row-Level Filtering, and Attributed-Based Masking should be utilized, as well as the recommended technique for handling large data volumes containing 100's of (around 1k ) PII columns.

Regard

Phanindra

Re: Masking techniques for more PII columns

BingWang — Wed, 18 Sep 2024 02:03:48 GMT

@Retired_mod

Thanks for the answers. To extend this question further, how can we apply column masks to DLT tables? DLT table is using Materialized Views under the hood. And databricks has mad coumn mask enabled for materialized view under Public Review. But same code not working in DLT table yet, any idea how or when DLT tables could apply column mask functions?

DLT tables column mask not working yet:

Regards

Bing

Re: Masking techniques for more PII columns

Meghla-C — Thu, 31 Oct 2024 23:00:28 GMT

Hello Bing ,
I was able to apply row-filter and column mask to DLT (both streaming tables and mv), it is still in Public preview though.

The DLT code and the function definitions are attached in the screenshots.

Re: Masking techniques for more PII columns

NandiniN — Fri, 01 Nov 2024 06:52:25 GMT

Agree with @Meghla-C , https://databricks.aha.io/ideas/ideas/DB-I-7941 this was the feature request and if you see the status, it is in preview.

Re: Masking techniques for more PII columns

BingWang — Mon, 04 Nov 2024 05:46:49 GMT

@Meghla-C Thanks for the update. Just to confirm, that the same source code to mask column which did not working before now works in DLT in my environment. Great job.