topic Re: Limitations for Unity Catalog on single user access mode clusters in Data Engineering

Limitations for Unity Catalog on single user access mode clusters

Nes_Hdr — Fri, 29 Nov 2024 09:38:54 GMT

Hello!

According to Databricks documentation on azure :

"On Databricks Runtime 15.3 and below, fine-grained access control on single user compute is not supported. Specifically:

You cannot access a table that has a row filter or column mask.
You cannot access dynamic views.
To read from any view, you must have SELECT on all tables and views that are referenced by the view."

Well, I am using DBR 15.4 LTS and getting the following error message when running an optimize job (OPTIMIZE table name) on some tables that have a row filter:

Py4JJavaError: An error occurred while calling o388.sql. : java.util.concurrent.ExecutionException: com.databricks.sql.managedcatalog.UnityCatalogServiceException: [RequestId=d5be0519-**** ErrorClass=INVALID_PARAMETER_VALUE.ROW_COLUMN_ACCESS_POLICIES_NOT_SUPPORTED_ON_ASSIGNED_CLUSTERS] Query on table **** with row filter or column mask not supported on assigned clusters.

This is my job cluster configuration:

Can you please help me figure out when could possible cause such an error ?

Thanks a lot!

Re: Limitations for Unity Catalog on single user access mode clusters

MuthuLakshmi — Fri, 29 Nov 2024 12:14:32 GMT

@Nes_Hdr Do you have Serverless enabled for your workspace?

Fine-grained access control on single user compute: Databricks Runtime 15.4 LTS does support fine-grained access control on single user compute, but it relies on serverless compute to run data filtering. This means that when a query accesses objects like tables with row filters or column masks, the single user compute resource passes the query to the serverless compute to execute the data filtering.
Error Message: The error message you are encountering, ROW_COLUMN_ACCESS_POLICIES_NOT_SUPPORTED_ON_ASSIGNED_CLUSTERS, indicates that the query on the table with row filters or column masks is not supported on assigned clusters. This suggests that the single user cluster you are using might not be configured to leverage serverless compute for these operations.

In the documentation it was mentioned that Serverless needs to be enabled in that workspace.
https://learn.microsoft.com/en-us/azure/databricks/compute/access-mode-limitations#assigned-limitations

Hope this helps!

Re: Limitations for Unity Catalog on single user access mode clusters

Nes_Hdr — Mon, 02 Dec 2024 09:38:42 GMT

@MuthuLakshmi Thank you very much for the reply!

Yes, I have serverless compute enabled on the workspace, but the cluster I am using is not serverless.
The error happens whenever access mode is "Single user" (see pictures below). It also happens when using DBR 16.

Re: Limitations for Unity Catalog on single user access mode clusters

Nes_Hdr — Thu, 05 Dec 2024 08:42:48 GMT

Can someone from Databricks check this out?

I am still confused why this doesn't work as it should.

Re: Limitations for Unity Catalog on single user access mode clusters

arzb — Tue, 10 Dec 2024 08:21:27 GMT

Hello,

We have the same issue. Please, any solution?

Re: Limitations for Unity Catalog on single user access mode clusters

arzb — Tue, 10 Dec 2024 09:31:10 GMT

Hello @Nes_Hdr ,

Do you have activated "serverless compute for workflows...." in the Databricks account?

Re: Limitations for Unity Catalog on single user access mode clusters

Nes_Hdr — Wed, 11 Dec 2024 09:16:02 GMT

@arzb yes, serverless copute is enabled

Re: Limitations for Unity Catalog on single user access mode clusters

arzb — Wed, 11 Dec 2024 09:23:35 GMT

Hi, then I don´t know what more we can do for fix this.

Please "Databricks" help

Re: Limitations for Unity Catalog on single user access mode clusters

MuthuLakshmi — Wed, 11 Dec 2024 09:36:25 GMT

@Nes_Hdr Does this OPTIMIZE works for some tables with row filter and fail for some?
Or did it never work in Single user?

Re: Limitations for Unity Catalog on single user access mode clusters

Nes_Hdr — Wed, 11 Dec 2024 09:58:57 GMT

@MuthuLakshmi it never worked on any table with a row filter using a Single user access mode cluster. The same query succeeds using a shared cluster.

The problem is reproducible, you would just need to create a table and apply a row filter function on it, then try to run Optimize table_name using a single user cluser (using DBR 15.4, and serverless is enabled in the workspace) ...

Re: Limitations for Unity Catalog on single user access mode clusters

MuthuLakshmi — Wed, 11 Dec 2024 10:38:50 GMT

@Nes_Hdr Single user compute uses fine-grained access control to access the tables with RLS/CLM enabled.

There is no specific details about OPTIMIZE being supported in Single user mode.

Under this doc limitations of FGAC mentions that

"No support for write or refresh table operations on tables that have row filters or column masks applied.

Specifically, DML operations, such as INSERT, DELETE, UPDATE, REFRESH TABLE, and MERGE, are not supported. You can only read (SELECT) from these tables."
https://docs.databricks.com/en/compute/single-user-fgac.html

Re: Limitations for Unity Catalog on single user access mode clusters

arzb — Wed, 29 Jan 2025 10:55:55 GMT

Hello,

And would be possible to do a "saveAsTable" with a table tha have column mask applied?

Thanks and regards

Re: Limitations for Unity Catalog on single user access mode clusters

arzb — Thu, 30 Jan 2025 11:00:22 GMT

@MuthuLakshmi , please any news about this?

Thanks

Re: Limitations for Unity Catalog on single user access mode clusters

robert_dillitz — Thu, 18 Sep 2025 09:42:49 GMT

@arzb @Nes_Hdr Command support for FGAC on dedicated compute was added in DBR 17.1: https://docs.databricks.com/aws/en/compute/single-user-fgac#support-for-ddl-show-describe-and-other-commands 🎉