topic Re: Row level Data MASKING on multiple tables in Data Governance

Row level Data MASKING on multiple tables

ChristophIRE — Wed, 14 Oct 2026 09:22:25 GMT

Hi Community,

I'm looking for guidance on implementing row-level masking (RLM) at scale in Databricks using Unity Catalog.

Our challenge is that we have schemas containing a large number of tables (for example, one schema contains approximately 120 tables). We would like to enforce a consistent masking policy based on a business identifier (e.g., Customer ID, Policy Number, Business Unit, Broker ID, etc.), where users can access the row but sensitive values are dynamically masked based on their entitlements.

The challenge is that not all tables contain the identifier required to determine the masking rule. Some tables contain the key directly, while others are lookup/reference tables or contain only related attributes.

A few questions:

What is the recommended approach for implementing row-level masking across a schema when the masking key is not present in every table?
Have others used entitlement, lookup, or reference tables to drive masking decisions in these scenarios?
Is there a scalable governance pattern for avoiding the need to create and maintain masking policies across 100+ tables?
How have you handled cases where the masking identifier exists only in a subset of tables and must be derived through relationships with other tables?
Are there any best practices using Unity Catalog masking policies, ABAC, governed views, or other governance features for this type of design?
Performance Consideration: Has anyone observed a significant performance impact when implementing row-level masking policies that rely on entitlement, lookup, or reference tables, particularly where the masking decision must be derived through joins rather than a security attribute being present on the target table?

Our objective is to implement a maintainable, centrally governed masking solution that can be applied consistently across large numbers of tables without introducing excessive operational overhead.

Current thinking: Use a central entitlement table to determine which users can view unmasked values. However, we are trying to understand whether Databricks would recommend:

Propagating the masking/security attribute into all tables requiring masking, or
Deriving masking decisions through centralized entitlement and reference tables.

We're particularly interested in the governance, maintainability, scalability, and performance trade-offs of these approaches in large enterprise implementations.

Any patterns, lessons learned, or architectural recommendations would be greatly appreciated.

Thanks!

Christopher

Re: Row level Data MASKING on multiple tables

data_pulse — Wed, 14 Oct 2026 10:28:42 GMT

@ChristophIRE

This is something I worked through recently on a similar use case: enforcing row-level security across multiple tables with a central entitlement model. For this scale, ABAC is the right primary governance pattern rather than maintaining individual masks/filters on 100+ tables.

1. Masking key not present in every table
Derive or materialise the security attribute in a governed serving layer, or use a governed view rather than making every policy discover it through complex joins inside the policy path. Keeps the policy logic simple and avoids runtime join overhead on every query.

2. Entitlement/lookup tables driving the masking decision
Yes, this is a documented Databricks pattern. A small entitlement/mapping table maps current_user() to allowed business attributes or masking levels, and the policy UDF performs the lookup at query time. Keep the table small and the lookup logic simple, a single equality predicate on user_email pushes cleanly to the parquet scan and stays Photon-native.

One hard constraint worth knowing upfront: the entitlement/mapping table must remain free of row filters or column masks. Unity Catalog does not allow a table with an active policy to be read inside another table’s policy, you’ll hit UNSUPPORTED_NESTED_ROW_OR_COLUMN_ACCESS_POLICY. Protect the mapping table using normal UC privileges instead, users never need direct access to it, the UDF runs under the owner's rights to read the mapping table, while current_user() still resolves to the calling user for the row filter itself.

3. Avoiding policy management on 100+ tables
Use tag-based ABAC policies applied via governed tags in Catalog Explorer. The policy automatically applies to every table/column carrying that tag, including newly tagged objects added later, no ALTER TABLE on each table required.

4. Identifier exists only in some tables
Avoid deriving it through arbitrary business-table joins inside the policy path. Propagate a stable security attribute into the governed layer where practical. Lookup/reference tables with no entity-level security relationship may not require the same policy at all.

5. ABAC vs masks vs views
A scalable pattern:

Governed Tags → Schema/Catalog ABAC → Policy UDF → Entitlement Mapping Table.

Use dynamic views where you need joins, transformations, or derived security attributes. Use table-level row filters/column masks for exceptions rather than as the primary mechanism across 100+ tables.

6. Performance
Row filters and column masks execute UDF logic at query time, so there is a runtime cost. A few observations from my own implementation on this pattern:

SQL UDFs inline at planning time: Photon sees a native BroadcastHashJoin, no ColumnarToRow transition. A Python UDF doing the same lookup would fall back on every row via BatchEvalPython.
The per-user predicate pushes to the parquet scan, so the broadcast payload stays small regardless of total table size.
Overhead scales with permitted access scope, not table row count: Tested up to ~18B rows table. Users with narrow access on larger tables outperformed broad-access users on smaller ones due to file pruning
Benchmark representative workloads: large entitlement tables or multi-table joins in the policy path will hurt performance noticeably

Recommendation

Propagate/materialise the security attribute where practical and use a central entitlement table for authorisation. Reserve runtime joins to reference/business tables for cases where deriving the attribute beforehand is genuinely impractical. For 100+ tables, tag-based ABAC with a single central UDF is the only maintainable path at that scale.

Have covered this pattern and the scaling observations in more detail in another community post where I mentioned bit more around scaling and worth going through it.

Re: Row level Data MASKING on multiple tables

Ashwin_DSA — Wed, 14 Oct 2026 11:27:44 GMT

Hi @ChristophIRE,

For your scenario of 120+ tables in a schema, the Databricks documentation explicitly recommends ABAC over per-table row filters and column masks. The core idea is that you define policies at the catalog or schema level, and they apply automatically to any table whose columns carry the right. You don't need to write or maintain a separate masking policy for each table. Refer to this.

Here is how it works... You basically create governed tags at the account level (e.g., a pii tag with values like ssn, email, name), then tag the sensitive columns across your tables. You write one ABAC policy scoped to your schema that uses MATCH COLUMNS has_tag_value('pii', 'email') to dynamically discover which columns to mask. When new tables are added and tagged, the policy applies automatically with no additional configuration.

Your main challenge, as I read it, is how to handle the tables when the masking key isn't there. There are a few patterns to consider.

For column masking (hiding PII values).... ABAC's MATCH COLUMNS clause is something to explore. If a table doesn't have a column tagged pii, the policy simply doesn't apply to that table, and no masking occurs. This is by design. You tag only the columns that actually contain sensitive data, and the policy targets them dynamically. Tables that are pure lookup/reference tables without PII columns are naturally excluded.

For row filtering (restricting which rows a user can see)... This is where things get more nuanced. If the business identifier needed to determine row-level access (e.g., Customer ID or Business Unit) isn't present in a given table, you have two realistic options:

Propagate the security attribute into the table. Add the filtering key (e.g., business_unit) as a column to tables that need row-level filtering. This is the more performant option because the row filter UDF can reference the column directly without any joins. It also enables predicate pushdown, which matters at scale.
Use a centralised entitlement/lookup table inside the UDF. You can write a UDF that checks a small mapping table to determine whether the current user has access. The Databricks docs provide an option, including support for conditional masking and even automatic access expiration.

Since you mentioned you are thinking about the entitlement pattern, here's how it works with ABAC. You create a small entitlement table that maps users to the data they're allowed to see:

CREATE TABLE access_rules ( principal VARCHAR(255), allowed_value VARCHAR(64) );

Then your UDF checks this table using session_user():

CREATE FUNCTION row_access_check(business_key STRING) RETURNS BOOLEAN RETURN EXISTS ( SELECT 1 FROM access_rules WHERE principal = session_user() AND allowed_value = business_key );

You can then attach this as an ABAC row filter policy scoped to your schema, using MATCH COLUMNS to dynamically find the business key column across tables. Here is some useful documentation.

You can refer to this page for details on the performance impact. In general try to keep lookup tables and opt for simple UDFs.

Refer to the best practices documented here.

Let us know if you hit any issues. Happy to help.

Re: Row level Data MASKING on multiple tables

balajij8 — Wed, 14 Oct 2026 17:59:07 GMT

The recommended approach for applying filtering and masking consistently across large numbers of tables is Attribute Based Access Control (ABAC) combined with governed tags. ABAC policies scale across catalogs and schemas and are defined by higher level admins, generally preventing table owners from overriding them. They are also evaluated more efficiently than per table UDFs. You can configure a single schema level policy (masking any column tagged pii = ssn) that automatically covers all 120 tables without having to write 120 separate ALTER TABLE SET MASK statements. Check the limitations here

On the query regarding whether to propagate the attribute or derive it through entitlement and reference tables, both patterns are supported but there are distinct performance trade offs. Using mapping tables inside policy UDFs such as EXISTS (SELECT 1 FROM map_table WHERE principal = session_user() AND attribute = row_value) is possible but use ABAC if feasible. If you must use external lookup tables, keep them small to trigger a broadcast hash join. Large lookup tables will force a slow shuffle join, and you should avoid complex joins or multi hop derivations inside UDFs. For high volume tables missing the masking key, propagating the governance attribute during the silver layer transformation is a good route for performance, while entitlement tables are fine if they are small and keyed on simple equality.

If you are utilizing a medallion architecture with Spark Declarative Pipelines, the right fit is to embed the security attribute during the silver layer transformation by denormalizing it from wherever the relationship natively lives. From there, tag the sensitive columns with governed tags and enforce the rules using one or two schema level ABAC policies. More details here