Feature request delta tables : drop duplicate rows

[MRTN]

A deltaTable.dropDuplicates(columns) would be a very nice feature, simplifying the complex procedures that are suggested online.

Or am I missing any existing procedures that can be done withouth merge operations or similar?

[Hubert-Dudek]

It would be helpful. Currently, the best way is just to read the table as a dataframe and use Pyspark dropDuplicates().

# Load the table
df = spark.table("yourtable")
 
# Drop duplicates based on the Id and Name columns
df = df.dropDuplicates(["Id", "Name"])
 
# Overwrite the original table with the resulting dataframe
df.write.mode("overwrite").saveAsTable("yourtable")

[DRGutierrez]

This worked perfectly, and much easier than all the complex solutions that are suggested online.

[Victor_D]

This is basically wiping and rewriting the whole table. Obviously it's a very easy solution, but very expensive.

There's a reason why the "usual" solutions are very complex, because they only target the duplicated rows.

[akshaybhan92]

is this still the best method?

[MRTN · Accepted solution]

I created a feature request in the delta table project: [Feature Request] data deduplication on existing delta table · Issue #1767 · delta-io/delta (github....