Row filtering and Column masking not working even when requirements the are met

[muano_makhokha]

I have been trying to use the Row filtering and Column masking feature to redacted columns and and filter rows based on the group a user is in.

I have all the necessary permissions and I've used cluster's with version 15.4 and higher.

When I run the following statements 

ALTER TABLE testtable_new ALTER COLUMN payload SET MASK payload_mask;

ALTER TABLE testtable_new SET ROW FILTER row_filter ON (payload);

 I get the this error

UnityCatalogServiceException: [RequestId=48187b09-7063-42e2-9b12-643717b0eb91 ErrorClass=ABORTED.UC_DBR_TRUST_VERSION_TOO_OLD] The Databricks Runtime being used no longer supports this operation. Please use the latest version (you may just need to restart your cluster).

[BigRoux · Accepted solution]

Here are some things to consider/try:

The UnityCatalogServiceException error you are encountering, ABORTED.UC_DBR_TRUST_VERSION_TOO_OLD, generally indicates that the Databricks Runtime (DBR) version you are using no longer supports the operation, such as setting row filters or column masks. While Databricks Runtime 15.4 LTS and higher theoretically supports these features, there are some known limitations and additional requirements you should be aware of:

### Causes and Insights: 1. Cluster Type and Compute Mode: - Certain operations like ALTER TABLE ... SET MASK or ALTER TABLE ... SET ROW FILTER are only supported on specific compute clusters. Dedicated clusters (formerly single-user) might not support these operations unless serverless compute is enabled. These operations may work on Standard compute mode (shared clusters) or Serverless SQL Warehouses.

2. Serverless Compute Requirement:

   • Even though DBR 15.4 LTS supports fine-grained access control in theory, the actual enforcement of row filters or column masks might require serverless compute to handle backend filtering and masking at runtime. Without enabling serverless compute in your workspace, attempting to set row filters or column masks on dedicated compute will result in errors.
3. Runtime Version:

   • If you restarted your cluster but are still encountering the issue, double-check that the cluster uses an officially-supported DBR version for your use case (15.4+). Sometimes, errors persist if there are inconsistencies in the runtime setup or cluster properties.
4. Expected Behavior and Limitations:

   • While SELECT queries on tables with existing row filters or column masks may work on dedicated compute (as they could rely on functionalities like Filtering Fleet for reads), DDL operations like ALTER TABLE are another matter. Filtering Fleet currently supports only read operations (queries or append operations), and DDL commands are not yet fully supported for tables with row filters or column masks on certain cluster types.

### Steps to Resolve: 1. Cluster Restart Verification: - Recreate the cluster on DBR 15.4+ and ensure it is operating in Standard mode or on a Serverless SQL Warehouse. If you are using a dedicated cluster, verify that the workspace is configured for serverless compute.

2. Serverless Compute Configuration:

   • Confirm that your workspace is enabled for serverless compute. You can do so by checking your workspace's administrative settings or consulting with your Databricks workspace admin.
3. Switch to Compatible Cluster Types:

   • If serverless compute is not enabled, try using either Standard compute or shared clusters to execute operations involving row filters or column masks.
4. Documentation Reference:

   • Review Databricks documentation on row filtering and column masking for Unity Catalog for further clarity on requirements and limitations: Databricks Documentation.

By ensuring the compute and cluster configuration meets these requirements, you should be able to overcome this error. If problems persist, it may be necessary to escalate this issue through your Databricks support channel, including specifics like the workspace ID, cluster configurations, and exact commands executed triggering the error.

Hope this help, Big Roux.