Prevent Duplicate Entries to enter to delta lake Storage
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2021 09:24 AM
I have a data frame and I write this data frame to adls table, next day I get an updated data frame which has some records from the past also and i want to update the delta table without creating duplicate
Labels:
- Labels:
-
Azure
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-25-2021 09:25 AM
This is a task for Merge command - you define condition for merge (your unique column) and then actions.
MERGE INTO target
USING src
ON target.column = source.column
WHEN MATCHED THEN
UPDATE SET *
WHEN NOT MATCHED
THEN INSERT *could be your dataframe registered as temporary view):