https://community.databricks.com/t5/get-started-discussions/remove-duplicate-records-using-pyspark/m-p/85932#M8197 <P>Hi <a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/25618">@sanjay</a>&nbsp;,</P><P>You can write window function that will rank your rows and then filter rows based on that rank.</P><P>Take a look on below stackoverflow thread:&nbsp;</P><P><A href="https://stackoverflow.com/questions/63343958/how-to-drop-duplicates-but-keep-first-in-pyspark-dataframe" target="_blank">https://stackoverflow.com/questions/63343958/how-to-drop-duplicates-but-keep-first-in-pyspark-dataframe</A></P><P>&nbsp;</P> Wed, 28 Aug 2024 18:36:19 GMT szymon_dybczak 2024-08-28T18:36:19Z https://community.databricks.com/t5/get-started-discussions/remove-duplicate-records-using-pyspark/m-p/85883#M8196 <P>Hi,</P><P>I am trying to remove duplicate records from pyspark dataframe and keep the latest one. But somehow df.dropDuplicates["id"] keeps the first one instead of latest. One of the option is to use pandas drop_duplicates, Is there any solution in pyspark.</P><P>Thanks,</P><P>Sanjay</P> Wed, 28 Aug 2024 17:15:01 GMT https://community.databricks.com/t5/get-started-discussions/remove-duplicate-records-using-pyspark/m-p/85883#M8196 sanjay 2024-08-28T17:15:01Z https://community.databricks.com/t5/get-started-discussions/remove-duplicate-records-using-pyspark/m-p/85932#M8197 <P>Hi <a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/25618">@sanjay</a>&nbsp;,</P><P>You can write window function that will rank your rows and then filter rows based on that rank.</P><P>Take a look on below stackoverflow thread:&nbsp;</P><P><A href="https://stackoverflow.com/questions/63343958/how-to-drop-duplicates-but-keep-first-in-pyspark-dataframe" target="_blank">https://stackoverflow.com/questions/63343958/how-to-drop-duplicates-but-keep-first-in-pyspark-dataframe</A></P><P>&nbsp;</P> Wed, 28 Aug 2024 18:36:19 GMT https://community.databricks.com/t5/get-started-discussions/remove-duplicate-records-using-pyspark/m-p/85932#M8197 szymon_dybczak 2024-08-28T18:36:19Z