topic Architectural Pattern for PII: "Redact at Ingestion" vs. "Dynamic Masking at Consumption in Data Engineering

Architectural Pattern for PII: "Redact at Ingestion" vs. "Dynamic Masking at Consumption

Khasim_1 — Sat, 19 Sep 2026 16:00:48 GMT

Hi everyone, we are defining our PII strategy in Unity Catalog. We are split on whether to: A) Redact/Hash PII at the Silver layer (permanent change), or B) Keep PII in Silver and use Dynamic Data Masking at the Gold/View layer.

Does "Redact at Ingestion" (A) cause too much friction for Data Science teams who occasionally need the raw data?
Is the Dynamic Masking approach (B) robust enough for strict audits (e.g., GDPR/CCPA)? Are there "escape hatches" where power users might accidentally bypass the masking via an API?
Are you using Unity Catalog’s built-in Data Classification tags to trigger these policies automatically?

Re: Architectural Pattern for PII: "Redact at Ingestion" vs. "Dynamic Masking at Cons

Ashwin_DSA — Sun, 20 Sep 2026 20:11:18 GMT

Hi @Khasim_1,

I would keep the PII in Silver within your regulatory and internal retention limits and control access with Unity Catalog's row filters and column masks, rather than redacting permanently. The reason is the one you already gave. A permanent change at Silver is a one-way door and hard to reverse, and the moment a business user has a legitimate future need for the raw value, you are stuck re-ingesting from source if it still exists. That data can be valuable, and you do not want to throw it away early.

On your first question, yes, redacting at ingestion tends to create real friction for data science. Teams that occasionally need the raw value for entity resolution, fraud work, or joining across systems cannot get it back, and hashing quietly breaks joins and distribution analysis unless you are very careful with consistent salting, which then limits its usefulness anyway. What usually happens is you end up setting up a separate raw pipeline for those teams, which is the duplication you were trying to avoid. Just to be clear, hashing or redaction at Silver is usually pseudonymisation, not anonymisation, so if the value can still be re-identified, it remains personal data under GDPR and does not take you out of scope. Only genuine anonymisation does that.

On the second question, dynamic masking with Unity Catalog is robust enough for GDPR and CCPA work, because it is enforced by the engine at the metastore rather than in a view someone has to remember to use. The same row filter or column mask applies whether the data is read through SQL, a BI tool, a notebook, JDBC or ODBC, or a Genie Agent, and every access is written to the audit log system table, which is usually what an auditor would be interested in.

To your question about escape hatches, the real one is not the SQL API, it is direct access to the underlying cloud storage. If someone can read the Delta files in the bucket, none of the masking runs, so lock down your external locations and storage credentials and make sure nobody is querying the files directly. Beyond that, run on compute that enforces fine-grained access control, keep tight control of the privileged principals who can alter or drop a policy or who own the masking function, and remember that masking is access control, not erasure. A right-to-be-forgotten request still needs a real delete, so keep that as a separate process.

On your third question, yes, and this is where it becomes scalable. Rather than attach masks and filters table by table, tag the sensitive columns and let one ABAC policy match the tag and apply the rule to every current and future table that carries it. Unity Catalog's data classification can scan and populate those tags for you, and the policy then acts on them. I would review the classification output rather than trust it blindly, but once it is set you define the rule once and it follows the tag.

For context, this is close to what I am doing right now in a regulated engagement. We implemented both row-level and column-level security for the data science teams entirely through ABAC, and the same policies apply automatically when they query those tables through Genie Agents. You define it once, and it holds no matter how the data is accessed, because UC resolves the entitlement centrally and only returns what the user is allowed to see. Per-table filters and masks work fine, but ABAC is far more scalable and easier to govern once you are past a handful of tables.

So my recommendation is to keep the raw data governed in Silver within your retention limits, define row and column security once with ABAC driven by classification tags, and lock down the storage layer. That gives data science the flexibility to reach raw data when they are entitled to it, and gives you an audit story that holds up.

One a side note, I do sometimes see data science teams take a cut from Bronze so they can work on truly raw, un-standardised data before any Silver cleansing. It can be a fair choice, but governance gets harder once that data sits in a separate environment, so if you go that way I would keep it inside the same Unity Catalog governance rather than outside it.

Hope this helps.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Re: Architectural Pattern for PII: "Redact at Ingestion" vs. "Dynamic Masking at Cons

gnakan — Sun, 27 Sep 2026 16:54:27 GMT

Hey @Khasim_1 , if it helps...did an experiment and blog post with results from the run that illustrates the pattern that @Ashwin_DSArecommended.

https://medium.com/lab-notes/redacting-pii-in-silver-leaves-it-in-bronze-column-masks-vs-redaction-on-databricks-f33175cf9267

Re: Architectural Pattern for PII: "Redact at Ingestion" vs. "Dynamic Masking at Cons

Islam_hoti — Mon, 28 Sep 2026 09:31:53 GMT

Hi,

In practice most teams end up with a hybrid: keep raw PII in one tightly restricted place, and use dynamic masking everywhere else.

1. Does redacting at Silver (A) create friction? Yes, often. Once PII is hashed or dropped in Silver, the raw data is gone for everyone. Data Science then asks for exceptions, and those turn into ad-hoc copies, which is worse for compliance. Two further points:

Hashing is not anonymization. Under GDPR, hashed identifiers are still pseudonymized personal data, so A doesn't take you out of scope.
Common middle ground: tokenize direct identifiers (email, SSN) in Silver with a consistent key, so joins still work. Keep the token → raw mapping in a separate restricted "PII vault" schema. Data Science works with tokens and asks for re-identification only when they need it.

2. Is dynamic masking (B) audit-proof? It can be. Masks and filters are enforced by Unity Catalog on every UC-governed access path (SQL Warehouses, notebooks, jobs, DLT, APIs), and every access is in system.access.audit. The real escape hatches are usually around UC, not through it:

Direct cloud storage access. Anyone with IAM/SAS access to the bucket or container can read the files and skip UC entirely. Lock storage down so only UC storage credentials can reach it.
Exempt users making copies. Anyone in an EXCEPT group can CREATE TABLE AS SELECT raw data into an unprotected table. Keep exempt groups small and alert on it through the audit log.
Policy admins. Users with MANAGE or ownership can change or drop policies. Separate policy admins from data engineers.
Right to erasure. Masking hides data but doesn't delete it. You still need a deletion process plus VACUUM to meet GDPR/CCPA erasure requests.

3. Data classification → automatic policies: Yes, this is the intended pattern. Data classification scans tables and tags PII columns. ABAC column-mask policies at catalog or schema level then match those tags (MATCH COLUMNS has_tag(...)), so new PII columns get masked without manual work. Tips:

Review classification results before relying on them fully. They are probabilistic, so false negatives happen.
Add manual governed tags where it matters.
Use a default-deny approach on sensitive schemas as a backstop.