Masking of PII data

ShankarM
Databricks Partner

We have a below requirement 

  • There is a history table where data need to be loaded incrementally.  
  • This table contains a PII field which has been masked using a custom masking function (allow visibility for a specific user group, XXXX for rest). 
  • When we run the ingestion job, we face an error saying cluster is not able to load data into the table where a column is masked. 

    Can you let us know what is the issue.