Databricks Community

You should use a MERGE command on this table to match records on the unique column. Delta Lake does not enforce primary keys so if you append only the duplicate ids will appear. Merge will provide you the functionality you desire. https://docs.databr...

This is a task for Merge command - you define condition for merge (your unique column) and then actions.MERGE INTO target USING src ON target.column = source.column WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *could be your dataf...

To achieve this you should use a merge command that will update rows that are existing with the unique ID. This will update the rows that already exist and insert the rows that do not. If you want to do it manually, you could delete rows using the DE...