Any Advice on Dynamic Masking while maintaining performance?

[tana_sakakimiya]

I plan to mask columns with a specific tag like "sensitive", "PII" which represents that the column values are ought to be seen by privileged user groups because they contain credentials or personal identity data.

To implement that i plan to create a function and apply to a catalog by policy.

However, I am worried on performance. did anyone try this and experience performance issue?
or is there anyone has better approach to perform the task?

Note that i have a requirement not to encrpt data.

Thank you in advance.

[saurabh18cs · Accepted solution]

Hi @tana_sakakimiya 

Your approach—using Unity Catalog column tags (like "sensitive" or "PII") and applying masking policies based on those tags—is a recommended and scalable way to manage data access in Databricks, especially for compliance and privacy. Masking policies are evaluated at query time, performance impact is minimal if logic is simple. only complex masking expressions involving udf's or regex may slow it down.

I would suggest to use ABAC (attribute based access control) which is coming soon already in private preview. ABAC allows you to control access to data based on attributes (tags, labels, or properties) of users, groups, or data objects, rather than just roles (RBAC). Dynamically evaluated and highly flexible. This approach avoid maintaining lot of roles with organisation changes.

An simple example for column masking rule under ABAC:

SET RULE analyst_sales_mask
ON CATALOG business_unit
COLUMN MASK mask_pii
TO `privileged_employees`
FOR TABLES
WHEN has_tag(‘txn’)
WHEN COLUMNS col_has_tag(‘pii’)

Br

Saurabh