topic Re: Error [UNAUTHORIZED_ACCESS]: Merging into a table with masked columns in Data Engineering

Error [UNAUTHORIZED_ACCESS]: Merging into a table with masked columns

thedatanerd — Wed, 17 Sep 2025 13:37:00 GMT

Hi,

I'm facing a weird error when trying to incrementally merge into a table with a masked column.

Table is external, but catalog is managed. Using Unity Catalog.

Error is:

[UNAUTHORIZED_ACCESS] Unauthorized access: PERMISSION_DENIED: Path-based access to table catalog.schema.table with row filter or column mask not supported. SQLSTATE: 42501

It is when doing:

merge_stats = ( 275 current_unity_delta_table.alias("current") 276 .merge(upsert_df.alias("upsert"), merge_condition) 277 .withSchemaEvolution() 278 .whenMatchedUpdateAll() 279 .whenNotMatchedInsertAll() --> 280 .execute() 281 )

Checking these posts 1,2 I tried changing the job cluster type to be Standard (formerly Shared), Dedicated (formerly: Single User) and also tried changing runtime to 17.2, 16.4,15.4 but nothing worked. Always throws same error.

Is there a tested and tried solution to this weird limitation? shouldn't Databricks handle this automatically?

Re: Error [UNAUTHORIZED_ACCESS]: Merging into a table with masked columns

K_Anudeep — Thu, 18 Sep 2025 15:11:08 GMT

Hello @thedatanerd ,

Tables with row filters or column masks can’t be accessed by path. Delta Lake’s object APIs (e.g., DeltaTable + .merge()) aren’t supported for these tables.This is expected by design .Run the merge via SQL against catalog.schema.table so Unity Catalog can enforce policies. Example: register your DataFrame as a temp view and do a MERGE INTO

Re: Error [UNAUTHORIZED_ACCESS]: Merging into a table with masked columns

thedatanerd — Thu, 18 Sep 2025 17:02:05 GMT

Thank you @K_Anudeep I was inspired by your comment and I tried to use DeltaTable.forName instead of DeltaTable.forPath and it worked.

Re: Error [UNAUTHORIZED_ACCESS]: Merging into a table with masked columns

K_Anudeep — Thu, 18 Sep 2025 17:10:59 GMT

Hello @thedatanerd ,

Thanks for the reply. Yes, you can access that by DeltaTable.forName if you know the table name or use MERGE into using SQL. If your question is answered,could you please accept this as a solution, as it would be useful to others in the community?

Thanks.