Managing Row-Level Security (RLS) vs. Views: Performance impacts at scale

[Khasim_1]

 Hi everyone, we are designing our security layer in Unity Catalog. We are debating between using Row-Level Security (RLS) predicates versus standard SQL Views for masking PII data.

1. When applying complex RLS predicates to tables with >100M rows, have you noticed a significant impact on query planning time compared to traditional view-based security?
2. Are there specific patterns you use to ensure RLS is applied consistently across both SQL Warehouse queries and DLT pipelines?
3. Is there a "limit" to the complexity of the predicate functions you’d recommend before switching to a materialized view strategy?

[ThomazNeto]

Hi Khasim,

Most of this is answered directly in the Databricks docs, so let me point to what they say.

1. Planning time vs. runtime. The docs don't describe a planning-time penalty; the cost is in what the optimizer is allowed to do. Both table-level row filters/column masks and ABAC policies insert a SecureView barrier in the plan, and predicates with side effects can't be pushed across it. The performance page calls this "the most common source of performance issues with protected tables" and says it "can also block partition pruning and liquid clustering optimizations, which can force full table scans", even "when the policy UDF resolves to a constant true". Simple equality and range predicates on the protected table still push down; predicates wrapped in functions or with implicit casts don't. The page shows EXPLAIN output where the predicate moves from PartitionFilters inside the scan to a Filter above SecureView, which is a good way to check your own queries.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/abac/performance

The recommendations on the overview page are short and worth following literally: use simple UDFs (prefer CASE over mapping tables or subqueries), limit the number of distinct masks on large tables, reduce UDF arguments, avoid row filters with many AND conjuncts, use deterministic expressions that can't throw (try_divide instead of /), and prefer SQL UDFs over Python. Identity functions like is_account_group_member are resolved once per query, not per row, so they're cheap; a lookup like EXISTS(SELECT 1 FROM access_table WHERE user = session_user()) is explicitly called out as slower. If you do use a mapping table, keep it small enough to broadcast, otherwise the optimizer falls back to a shuffle join. The docs also recommend testing on at least 1 million rows before production.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/filters-and-masks/

2. Consistency across SQL Warehouse and pipelines. Filters and masks are table-level, so enforcement is the same on a SQL warehouse, standard access mode (DBR 12.2 LTS+) and dedicated access mode (DBR 15.4 LTS+ with serverless enabled for the workspace).
https://docs.databricks.com/aws/en/data-governance/unity-catalog/filters-and-masks/manually-apply

For Lakeflow pipelines, streaming tables and materialized views declare them in the table definition: WITH ROW FILTER and the MASK column clause in CREATE OR REFRESH STREAMING TABLE, or the row_filter argument plus the DDL schema string in the Python decorator. Two details from the docs matter here: during a pipeline refresh the filter and mask functions run with the pipeline owner's rights (CURRENT_USER / IS_MEMBER evaluate as the owner), and only at query time as the invoker; and pipelines can call existing UC UDFs but cannot create them, so the functions must exist before the pipeline runs. ALTER TABLE is not allowed on streaming tables, so changes go through CREATE OR REFRESH.
https://docs.databricks.com/aws/en/ldp/unity-catalog
https://docs.databricks.com/aws/en/ldp/developer/ldp-sql-ref-create-streaming-table
https://docs.databricks.com/aws/en/ldp/developer/ldp-python-ref-table

3. Complexity limit. The docs don't give a threshold, and the alternative they point to isn't a materialized view. The recommendation is: when you need the same filtering or masking across many tables, use ABAC policies rather than per-table UDFs (the docs say policy logic is evaluated more efficiently than table-specific UDFs, and users covered by an EXCEPT clause skip the UDF entirely); when you need a curated, joined or reshaped version of the data for users who don't have access to the base tables, use a dynamic view. Before committing to filters/masks, also check the limitations list: time travel does not work on tables with row filters or column masks, clones are not supported, and MERGE does not work when the policy contains nesting, aggregations, windows, limits or non-deterministic functions.
https://docs.databricks.com/aws/en/data-governance/unity-catalog/filters-and-masks/
https://docs.databricks.com/aws/en/data-governance/unity-catalog/abac/abac-vs-rls-cm

Hope this helps.

[KarinDatabricks]

I advise RLS because as Tomaz said: you cannot apply time travel with masks or filters, also clones are not supported and MERGE does not work with them either. 

[Islam_hoti · Accepted solution]

Hi,

1. Planning time vs. views: Planning overhead is usually small. Row filters and column masks are inlined into the plan much like a view, and identity functions such as is_account_group_member() are resolved once per query, not per row. On 100M+ row tables, the real cost is usually at execution, from a SecureView barrier that the policy adds:

Simple predicates (col = 'x', range checks) still push down, so partition pruning and liquid clustering keep working.
Predicates that wrap a column in a function (e.g. date_format(col, ...)) or add an implicit cast are blocked by the barrier and can cause a full scan. This happens even if the filter returns true.
Use EXCEPT for exempt groups. That removes the barrier for them completely.

Always compare EXPLAIN output with and without the policy.

2. Consistency across SQL Warehouses and DLT: Filters and masks live on the table in Unity Catalog, so every engine sees the same rules. That's the main advantage over views, which users can bypass if they have access to the base table. Practical patterns:

Use ABAC policies + governed tags at catalog or schema level instead of setting filters table by table. New tables are covered once they're tagged.
In Lakeflow/DLT, define ROW FILTER / MASK on streaming tables and materialized views, or rely on tag-based ABAC.
Read protected tables from serverless compute or a supported DBR version (dedicated clusters need fine-grained access control support).
Check in CI with SHOW EFFECTIVE POLICIES.

3. When is a predicate "too complex"? There's no hard limit. Consider precomputing (e.g. per-region materialized views or pre-extracted columns) when:

the UDF joins to a lookup table too large to broadcast,
you need Python UDFs, regex on large text/JSON fields, or nested subqueries,
you apply many different masks on one table (reuse one function where you can).

Keep UDFs to simple SQL CASE / boolean logic, deterministic and error-safe (try_cast, try_divide). Test on at least 1M rows with realistic queries before production.