Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-13-2025 08:54 AM - edited 05-13-2025 08:56 AM
This error occurs; when we try to update all the cells of target_data without a single updated record in source_data(updates_data) , to resolve this issue add a update_time column with unix timestamp (or) make changes in at least one cell of streaming/batch/incremental data, so that the DeltaTable knows it's not a duplicate.
In your scenario when you re-run the notebook with current timestamp it picks only in hours and days not in seconds and minutes which makes the whole data as duplicate, since your ran within an hour or less then 60 minutes.
Lokesh Manne