topic Prevent Duplicate Entries to enter to delta lake Storage in Data Engineering

Prevent Duplicate Entries to enter to delta lake Storage

User16826994223 — Fri, 25 Jun 2021 16:24:13 GMT

I have a data frame and I write this data frame to adls table, next day I get an updated data frame which has some records from the past also and i want to update the delta table without creating duplicate

Re: Prevent Duplicate Entries to enter to delta lake Storage

User16826994223 — Fri, 25 Jun 2021 16:25:07 GMT

This is a task for Merge command - you define condition for merge (your unique column) and then actions.

MERGE INTO target
USING  src
ON target.column = source.column
WHEN MATCHED THEN
  UPDATE SET *
WHEN NOT MATCHED
  THEN INSERT *

could be your dataframe registered as temporary view):