Drop duplicates in Table

ChristianHofste
New Contributor II

Hi,

there is a function to delete data from a Delta Table:

deltaTable = DeltaTable.forPath(spark, "/data/events/")
deltaTable.delete(col("date") < "2017-01-01")

But is there also a way to drop duplicates somehow? Like deltaTable.dropDuplicates()...

I don't want to read the whole table as dataframe, drop the duplicates, and write it to storage again

shyam_9
Databricks Employee
Databricks Employee

Hi @Christian Hofstetter,

You can check here for info on the same,

https://docs.delta.io/0.4.0/delta-update.html#data-deduplication-when-writing-into-delta-tables