Hi,

I'm facing a weird error when trying to incrementally merge into a table with a masked column.

Table is external, but catalog is managed. Using Unity Catalog.

Error is:

[UNAUTHORIZED_ACCESS] Unauthorized access: PERMISSION_DENIED: Path-based access to table catalog.schema.table with row filter or column mask not supported. SQLSTATE: 42501

It is when doing:

merge_stats = (
    275     current_unity_delta_table.alias("current")
    276     .merge(upsert_df.alias("upsert"), merge_condition)
    277     .withSchemaEvolution()
    278     .whenMatchedUpdateAll()
    279     .whenNotMatchedInsertAll()
--> 280     .execute()
    281 )

Checking these posts 1,2 I tried changing the job cluster type to be Standard (formerly Shared), Dedicated (formerly: Single User) and also tried changing runtime to 17.2, 16.4,15.4 but nothing worked. Always throws same error.

Is there a tested and tried solution to this weird limitation? shouldn't Databricks handle this automatically?

Path based access not supported for tables with row filters? 

path-based access to a table with row filters or column masks is not supported