03-28-2019 08:06 AM
You can use dropDuplicates
https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=distinct#pyspark.sql.Data...
