topic Re: Requirement to remove/skip column(s) in the downstream tables/views while PII data masking in Data Engineering

Requirement to remove/skip column(s) in the downstream tables/views while PII data masking

RiyazAliM — Wed, 08 Jan 2025 03:04:23 GMT

Hi there,

As a compliance measure, I'm tasked with masking the PII data starting from bronze to silver and all the tables and views downstream. I suggested my clients to use row filters and column masks as mentioned in the doc.

However, when a user who doesn't have access to a particular tables queries it, he SHOULD NOT be able to see any of the PII column in that table (not masked, the columns should be invisible to non-compliant users). But if a user with access queries it, those columns should be visible to him.

Any idea how to achieve this?

Thanks in advance.

Re: Requirement to remove/skip column(s) in the downstream tables/views while PII data masking

Walter_C — Wed, 08 Jan 2025 12:04:38 GMT

Is there a reason for this column to not to be masked, the main reason of this mask operation is to hide the columns for non compliant users, so the only option will me to mask it.

Re: Requirement to remove/skip column(s) in the downstream tables/views while PII data masking

RiyazAliM — Thu, 09 Jan 2025 12:33:15 GMT

Hi @Walter_C,

When you mask a column, the column will still be visible, but the data in that masked column will be encrypted (or masked) right?
My requirement is that, the non compliant user, should never know that the particular masked column exists in the first place.

Thank you,

Re: Requirement to remove/skip column(s) in the downstream tables/views while PII data masking

Walter_C — Thu, 09 Jan 2025 12:57:14 GMT

You are right, on this case we might need to open a feature request through https://docs.databricks.com/en/resources/ideas.html#ideas