Show all distinct values per column in dataframe Problem Statement:I want to see all the distinct values per column for my entire table, but a SQL que...

[Artem_Y]

Show all distinct values per column in dataframe

Problem Statement:

I want to see all the distinct values per column for my entire table, but a SQL query with a collect_set() on every column is not dynamic and too long to write.

Use this code to show the output below:

%python
 
from pyspark.sql.functions import col, collect_set
 
distincts = df.agg(*(collect_set(col(c)).alias(c) for c in df.columns))
distincts.display()

[Anonymous]

@Artem Yevtushenko​ - This is great! Thank you for sharing!