Drop duplicates in Table
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-07-2020 04:21 AM
Hi,
there is a function to delete data from a Delta Table:
deltaTable = DeltaTable.forPath(spark, "/data/events/")
deltaTable.delete(col("date") < "2017-01-01")
But is there also a way to drop duplicates somehow? Like deltaTable.dropDuplicates()...
I don't want to read the whole table as dataframe, drop the duplicates, and write it to storage again
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-19-2020 02:55 PM
Hi @Christian Hofstetter,
You can check here for info on the same,https://docs.delta.io/0.4.0/delta-update.html#data-deduplication-when-writing-into-delta-tables