This issue occurs because Databricks does not support applying row filters or column masks to external tables when path-based access is used. While you are able to set the row filter policy on your table with no immediate error, the limitation only becomes apparent when an operation such as recreating the table via DBT (which relies on path-based access due to the external table configuration) is attempted. This triggers the error:

INVALID_PARAMETER_VALUE.PATH_BASED_ACCESS_NOT_SUPPORTED_FOR_TABLES_WITH_ROW_COLUMN_ACCESS_POLICIES

Explanation of the Error

• Row filters/column masks introduce additional metadata and security management that Databricks expects to be fully controlled. Managed tables provide this capability because the system governs both storage and access, allowing Databricks to enforce row/column-level policies reliably.

• External tables, in contrast, are defined with a location_root and rely on underlying storage path-based access. When you apply a row filter to such tables, Databricks cannot guarantee enforcement using only storage-level permissions, leading to a conflict.

• The actual creation of the row filter succeeds because Databricks allows attaching access policies to any table initially, but enforcement (and validation) only occurs when the table is accessed in a manner that checks all associated policies, such as reading, recreating, or querying via DBT.

Why No Error When Setting the Policy

• Setting the policy updates table metadata but does not test full compatibility with future access patterns.

• Running the DBT job typically triggers a table recreation or re-write, invoking deeper compatibility checks. If path-based access is required (external tables), and a row filter is present, Databricks blocks the operation with the above error.

Solutions and Workarounds

Here are your main options:

1. Convert the Table to Managed

• Switch from an external to a managed table so Databricks can safely enforce row filters and column masks.

• Pros: Fully supported by Databricks; access policies work seamlessly.

• Cons: You lose some storage flexibility provided by external tables.

2. Remove Row-Level Access Policies on External Tables

• Drop the row filter or column mask on the table if you must use path-based access.

• Pros: Allows you to use external tables and DBT normally.

• Cons: You lose fine-grained access control at the row/column level.

3. Use Views for Row Filtering

• Create views on top of your external tables to implement row-level security logic as part of the view definition.

• This avoids putting access policies directly on the table.

• Pros: Keeps the external table unchanged and applies row filtering through views.

• Cons: Slightly less elegant and policies are not enforced at the table metadata level.

4. Managed Table + Storage Location

• If you need more control over storage, consider using managed tables with external locations in Unity Catalog (if available). This blends the benefits of managed enforcement with external storage options.

5. Custom Security Enforcement

• Apply row-level security in your data pipelines or application logic rather than relying on Databricks metadata policies.

• Pros: Maximal compatibility.

• Cons: More operational overhead and possible security risks.

Summary Table

Recommendations

• The error is by design: Databricks prevents path-based access when row/column access policies are set to avoid security loopholes.

• For DBT workflows requiring external tables and row/column policies, consider replacing row-access policies with SQL views or restructuring your workflow to use managed tables where possible.