balajij8
Esteemed Contributor II

Hi @AanchalSoni 

Primary & Foreign Keys in Unity Catalog are Informational Only. Databricks does not enforce uniqueness during writes to avoid the massive performance overhead. You are responsible for maintaining data integrity in the pipeline logic. Table definition is just the blueprint for documentation and optimization. You can use below

  • MERGE: Instead of INSERT, use MERGE to upsert records as its the standard way to prevent duplicates in Delta Lake.
  • Deduplication: Use row number, over & partition by (over (partition by id order by timestamp desc) = 1) in your code before writing to the tables
  • Check Constraints: While Primary & Foreign Keys are not enforced, NOT NULL and CHECK constraints are strictly enforced.