Hi @ChristophIRE,
For your scenario of 120+ tables in a schema, the Databricks documentation explicitly recommends ABAC over per-table row filters and column masks. The core idea is that you define policies at the catalog or schema level, and they apply automatically to any table whose columns carry the right. You don't need to write or maintain a separate masking policy for each table. Refer to this.
Here is how it works... You basically create governed tags at the account level (e.g., a pii tag with values like ssn, email, name), then tag the sensitive columns across your tables. You write one ABAC policy scoped to your schema that uses MATCH COLUMNS has_tag_value('pii', 'email') to dynamically discover which columns to mask. When new tables are added and tagged, the policy applies automatically with no additional configuration.
Your main challenge, as I read it, is how to handle the tables when the masking key isn't there. There are a few patterns to consider.
For column masking (hiding PII values).... ABAC's MATCH COLUMNS clause is something to explore. If a table doesn't have a column tagged pii, the policy simply doesn't apply to that table, and no masking occurs. This is by design. You tag only the columns that actually contain sensitive data, and the policy targets them dynamically. Tables that are pure lookup/reference tables without PII columns are naturally excluded.
For row filtering (restricting which rows a user can see)... This is where things get more nuanced. If the business identifier needed to determine row-level access (e.g., Customer ID or Business Unit) isn't present in a given table, you have two realistic options:
-
Propagate the security attribute into the table. Add the filtering key (e.g., business_unit) as a column to tables that need row-level filtering. This is the more performant option because the row filter UDF can reference the column directly without any joins. It also enables predicate pushdown, which matters at scale.
-
Use a centralised entitlement/lookup table inside the UDF. You can write a UDF that checks a small mapping table to determine whether the current user has access. The Databricks docs provide an
option, including support for conditional masking and even automatic access expiration.
Since you mentioned you are thinking about the entitlement pattern, here's how it works with ABAC. You create a small entitlement table that maps users to the data they're allowed to see:
CREATE TABLE access_rules (
principal VARCHAR(255),
allowed_value VARCHAR(64)
);
Then your UDF checks this table using session_user():
CREATE FUNCTION row_access_check(business_key STRING) RETURNS BOOLEAN
RETURN EXISTS (
SELECT 1 FROM access_rules
WHERE principal = session_user() AND allowed_value = business_key
);
You can then attach this as an ABAC row filter policy scoped to your schema, using MATCH COLUMNS to dynamically find the business key column across tables. Here is some useful documentation.
You can refer to this page for details on the performance impact. In general try to keep lookup tables and opt for simple UDFs.
Refer to the best practices documented here.
Let us know if you hit any issues. Happy to help.
Regards,
Ashwin | Delivery Solution Architect @ Databricks
Helping you build and scale the Data Intelligence Platform.
***Opinions are my own***
