It would be helpful. Currently, the best way is just to read the table as a dataframe and use Pyspark dropDuplicates().

# Load the table
df = spark.table("yourtable")
 
# Drop duplicates based on the Id and Name columns
df = df.dropDuplicates(["Id", "Name"])
 
# Overwrite the original table with the resulting dataframe
df.write.mode("overwrite").saveAsTable("yourtable")

This worked perfectly, and much easier than all the complex solutions that are suggested online.

This is basically wiping and rewriting the whole table. Obviously it's a very easy solution, but very expensive.

There's a reason why the "usual" solutions are very complex, because they only target the duplicated rows.

is this still the best method?