Comment
02-12-2025
01:46 PM
02-12-2025
01:46 PM
Great post!
I went after more reading while reading each topics and I would like to add a few things here
1. Anonymization
I wouldn't use uuid() like this.
Using a hashing function would be better to ensure consistency across multiple runs.
F.sha2(F.concat(F.col("user_id"), F.lit(SALT)), 256)
3. Data Masking with Databricks SQL and Delta Lake
Now databricks launched better ways to do it using functions and unit catalog for column mask and row filters.
ALTER TABLE employees ALTER COLUMN ssn SET MASK mask_ssn;
ALTER TABLE employees SET ROW FILTER team_filter ON (current_user(), manager_id);
Thanks!