Show all distinct values per column in dataframe Problem Statement:I want to see all the distinct values per column for my entire table, but a SQL que...

Artem_Y — Thu, 14 Oct 2021 00:45:06 GMT

Show all distinct values per column in dataframe

Problem Statement:

I want to see all the distinct values per column for my entire table, but a SQL query with a collect_set() on every column is not dynamic and too long to write.

Use this code to show the output below:

%python
 
from pyspark.sql.functions import col, collect_set
 
distincts = df.agg(*(collect_set(col(c)).alias(c) for c in df.columns))
distincts.display()

Re: Show all distinct values per column in dataframe Problem Statement:I want to see all the distinct values per column for my entire table, but a SQL que...

Anonymous — Thu, 14 Oct 2021 17:25:19 GMT

@Artem Yevtushenko - This is great! Thank you for sharing!

topic Show all distinct values per column in dataframe Problem Statement:I want to see all the distinct values per column for my entire table, but a SQL que... in Data Engineering

Show all distinct values per column in dataframe Problem Statement:I want to see all the distinct values per column for my entire table, but a SQL que...

Re: Show all distinct values per column in dataframe Problem Statement:I want to see all the distinct values per column for my entire table, but a SQL que...