topic Re: how to get unique values of a column in pyspark dataframe in Data Engineering

how to get unique values of a column in pyspark dataframe

satya — Thu, 08 Sep 2016 07:01:18 GMT

like in pandas I usually do df['columnname'].unique()

Re: how to get unique values of a column in pyspark dataframe

raela — Wed, 05 Apr 2017 00:07:48 GMT

df.select("columnname").distinct().show()

Re: how to get unique values of a column in pyspark dataframe

ShuminWu — Thu, 15 Jun 2017 03:14:59 GMT

Hi, tried using .distinct().show() as advised, but am getting the error TypeError: 'DataFrame' object is not callable.

The dataframe was read in from a csv file using spark.read.csv, other functions like describe works on the df. any reason for this? how should I go about retrieving the list of unique values in this case?

sorry if question is very basic. noob at this. Thanks!

Re: how to get unique values of a column in pyspark dataframe

AbhishekYada — Mon, 17 Dec 2018 01:14:43 GMT

this code returns data that's not iterable, i.e. I see the distinct data bit am not able to iterate over it in code. Any other way that enables me to do it. I tried using toPandas() to convert in it into Pandas df and then get the iterable with unique values. However, running into '' Pandas not found' error message. How can I install Pandas i my pyspark env, if my local already has Pandas running!

Re: how to get unique values of a column in pyspark dataframe

Rodneyjoyce — Mon, 22 Apr 2019 11:18:21 GMT

To get the count of the distinct values:

df.select(F.countDistinct("colx")).show()

Or to count the number of records for each distinct value:

df.groupBy("colx").count().orderBy().show()

Re: how to get unique values of a column in pyspark dataframe

ldfo — Wed, 01 Jul 2020 22:05:32 GMT

Hi, this worked for me.

distinct_ids = [x.id for x in data.select('id').distinct().collect()]

Re: how to get unique values of a column in pyspark dataframe

Ger_Martinez — Wed, 02 Sep 2020 15:03:03 GMT

nice way, also very "pythonic" minded

Re: how to get unique values of a column in pyspark dataframe

AnujGupta — Sun, 22 Nov 2020 21:36:11 GMT

Thanks for this. The latter worked well for me. However, sorry for my ignorance here but what is F in the first one? The code works without the F. !The latter worked well for me.

Re: how to get unique values of a column in pyspark dataframe

AbimaelDomingue — Sat, 07 Aug 2021 05:43:41 GMT

If you want to use the values to make some processing, this is the way to go.

Re: how to get unique values of a column in pyspark dataframe

AbimaelDomingue — Sat, 07 Aug 2021 05:45:16 GMT

If you just want to print the results and not use the results for other processing, this is the way to go.

Re: how to get unique values of a column in pyspark dataframe

AbimaelDomingue — Sat, 07 Aug 2021 05:46:12 GMT

If you want to use the results for some other data processing this is the way to go.