Limitations for Unity Catalog on single user access mode clusters

[Nes_Hdr]

Hello! 

According to Databricks documentation on azure :

"On Databricks Runtime 15.3 and below, fine-grained access control on single user compute is not supported. Specifically:

• You cannot access a table that has a row filter or column mask.
• You cannot access dynamic views.
• To read from any view, you must have SELECT on all tables and views that are referenced by the view."

Well, I am using DBR 15.4 LTS and getting the following error message when running an optimize job (OPTIMIZE table name) on some tables that have a row filter:

Py4JJavaError: An error occurred while calling o388.sql. : java.util.concurrent.ExecutionException: com.databricks.sql.managedcatalog.UnityCatalogServiceException: [RequestId=d5be0519-**** ErrorClass=INVALID_PARAMETER_VALUE.ROW_COLUMN_ACCESS_POLICIES_NOT_SUPPORTED_ON_ASSIGNED_CLUSTERS] Query on table **** with row filter or column mask not supported on assigned clusters.

This is my job cluster configuration: 

Can you please help me figure out when could possible cause such an error ? 

Thanks a lot! 

[MuthuLakshmi]

@Nes_Hdr Do you have Serverless enabled for your workspace?

1. Fine-grained access control on single user compute: Databricks Runtime 15.4 LTS does support fine-grained access control on single user compute, but it relies on serverless compute to run data filtering. This means that when a query accesses objects like tables with row filters or column masks, the single user compute resource passes the query to the serverless compute to execute the data filtering.
2. Error Message: The error message you are encountering, ROW_COLUMN_ACCESS_POLICIES_NOT_SUPPORTED_ON_ASSIGNED_CLUSTERS, indicates that the query on the table with row filters or column masks is not supported on assigned clusters. This suggests that the single user cluster you are using might not be configured to leverage serverless compute for these operations.
   
   In the documentation it was mentioned that Serverless needs to be enabled in that workspace. 
   https://learn.microsoft.com/en-us/azure/databricks/compute/access-mode-limitations#assigned-limitati...
   
   Hope this helps! 
   
   
   

[Nes_Hdr]

@MuthuLakshmi Thank you very much for the reply! 

Yes, I have serverless compute enabled on the workspace, but the cluster I am using is not serverless. 
The error happens whenever access mode is "Single user" (see pictures below). It also happens when using DBR 16.

 

[Nes_Hdr]

Can someone from Databricks check this out? 

I am still confused why this doesn't work as it should.

[arzb]

Hello,

We have the same issue. Please, any solution?

[arzb]

Hello @Nes_Hdr ,

Do you have activated "serverless compute for workflows...." in the Databricks account?

[Nes_Hdr]

@arzb yes, serverless copute is enabled 

[arzb]

Hi, then I don´t know what more we can do for fix this.

Please "Databricks" help

[MuthuLakshmi]

@Nes_Hdr Does this OPTIMIZE works for some tables with row filter and fail for some?
Or did it never work in Single user?

[Nes_Hdr]

@MuthuLakshmi it never worked on any table with a row filter using a Single user access mode cluster. The same query succeeds using a shared cluster. 

The problem is reproducible, you would just need to create a table and apply a row filter function on it, then try to run Optimize table_name using a single user cluser (using DBR 15.4, and serverless is enabled in the workspace) ... 

[MuthuLakshmi]

@Nes_Hdr Single user compute uses fine-grained access control to access the tables with RLS/CLM enabled.

There is no specific details about OPTIMIZE being supported in Single user mode. 

Under this doc limitations of FGAC mentions that 

"No support for write or refresh table operations on tables that have row filters or column masks applied.

Specifically, DML operations, such as INSERT, DELETE, UPDATE, REFRESH TABLE, and MERGE, are not supported. You can only read (SELECT) from these tables."
https://docs.databricks.com/en/compute/single-user-fgac.html