topic How to take distinct of multiple columns ( &gt; than 2 columns) in pyspark datafarme ? in Data Engineering https://community.databricks.com/t5/data-engineering/how-to-take-distinct-of-multiple-columns-gt-than-2-columns-in/m-p/28220#M20043 <P></P> <P>I have 10+ columns and want to take distinct rows by multiple columns into consideration. How to achieve this using pyspark dataframe functions ? </P> <P></P> Tue, 05 Mar 2019 07:58:17 GMT srchella 2019-03-05T07:58:17Z How to take distinct of multiple columns ( > than 2 columns) in pyspark datafarme ? https://community.databricks.com/t5/data-engineering/how-to-take-distinct-of-multiple-columns-gt-than-2-columns-in/m-p/28220#M20043 <P></P> <P>I have 10+ columns and want to take distinct rows by multiple columns into consideration. How to achieve this using pyspark dataframe functions ? </P> <P></P> Tue, 05 Mar 2019 07:58:17 GMT https://community.databricks.com/t5/data-engineering/how-to-take-distinct-of-multiple-columns-gt-than-2-columns-in/m-p/28220#M20043 srchella 2019-03-05T07:58:17Z Re: How to take distinct of multiple columns ( > than 2 columns) in pyspark datafarme ? https://community.databricks.com/t5/data-engineering/how-to-take-distinct-of-multiple-columns-gt-than-2-columns-in/m-p/28221#M20044 <P></P> <P>You can use dropDuplicates</P> <P> <A href="https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=distinct#pyspark.sql.DataFrame.dropDuplicates" target="test_blank">https://spark.apache.org/docs/latest/api/python/pyspark.sql.html?highlight=distinct#pyspark.sql.DataFrame.dropDuplicates</A></P> <P></P> Thu, 28 Mar 2019 15:06:05 GMT https://community.databricks.com/t5/data-engineering/how-to-take-distinct-of-multiple-columns-gt-than-2-columns-in/m-p/28221#M20044 Sandeep 2019-03-28T15:06:05Z