gchandra
Databricks Employee
Databricks Employee

‎09-23-2024 06:59 AM - edited ‎09-23-2024 07:00 AM

Please try this pyspark solution. 
df = spark.table("yourtablename")
df = df.dropDuplicates()

or use the Row_number() query and use DELETE statement where rn > 1



~

View solution in original post