Drop duplicates in Table

ChristianHofste — Thu, 07 May 2020 11:21:37 GMT

Hi,

there is a function to delete data from a Delta Table:

deltaTable = DeltaTable.forPath(spark, "/data/events/")
deltaTable.delete(col("date") < "2017-01-01")

But is there also a way to drop duplicates somehow? Like deltaTable.dropDuplicates()...

I don't want to read the whole table as dataframe, drop the duplicates, and write it to storage again

Re: Drop duplicates in Table

shyam_9 — Tue, 19 May 2020 21:55:34 GMT

Hi @Christian Hofstetter,

You can check here for info on the same,