You should use a MERGE command on this table to match records on the unique column. Delta Lake does not enforce primary keys so if you append only the duplicate ids will appear.
Merge will provide you the functionality you desire.
https://docs.databricks.com/spark/latest/spark-sql/language-manual/delta-merge-into.html
