Error [UNAUTHORIZED_ACCESS]: Merging into a table with masked columns

[thedatanerd]

Hi,

I'm facing a weird error when trying to incrementally merge into a table with a masked column.

Table is external, but catalog is managed. Using Unity Catalog.

Error is:

[UNAUTHORIZED_ACCESS] Unauthorized access: PERMISSION_DENIED: Path-based access to table catalog.schema.table with row filter or column mask not supported. SQLSTATE: 42501

It is when doing:

merge_stats = (
    275     current_unity_delta_table.alias("current")
    276     .merge(upsert_df.alias("upsert"), merge_condition)
    277     .withSchemaEvolution()
    278     .whenMatchedUpdateAll()
    279     .whenNotMatchedInsertAll()
--> 280     .execute()
    281 )

Checking these posts 1,2 I tried changing the job cluster type to be Standard (formerly Shared), Dedicated (formerly: Single User) and also tried changing runtime to 17.2, 16.4,15.4 but nothing worked. Always throws same error.

Is there a tested and tried solution to this weird limitation? shouldn't Databricks handle this automatically?

Path based access not supported for tables with row filters? 

path-based access to a table with row filters or column masks is not supported 

[K_Anudeep]

Hello @thedatanerd ,

Tables with row filters or column masks can’t be accessed by path. Delta Lake’s object APIs (e.g., DeltaTable + .merge()) aren’t supported for these tables.This is expected by design .Run the merge via SQL against catalog.schema.table so Unity Catalog can enforce policies. Example: register your DataFrame as a temp view and do a MERGE INTO

[thedatanerd · Accepted solution]

Thank you @K_Anudeep I was inspired by your comment and I tried to use DeltaTable.forName instead of DeltaTable.forPath and it worked.

[K_Anudeep]

Hello @thedatanerd ,

Thanks for the reply. Yes, you can access that by DeltaTable.forName if you know the table name or use MERGE into using SQL. If your question is answered,could you please accept this as a solution, as it would be useful to others in the community?

Thanks.