Unforgiven
Valued Contributor III

https://sparkbyexamples.com/pyspark/pyspark-distinct-to-drop-duplicates/

refer this link above may match with your concern. hope this can make and help in this case