How DISTINCT Works in Spark

When you apply DISTINCT, Spark performs a shuffle to eliminate duplicate rows. This involves:

1. Sorting the data or grouping it to identify unique rows.
2. Moving data across partitions (shuffle) based on its content.
3. Writing intermediate results to memory/disk.

If your dataset is large (30GB compressed CSV), the amount of shuffle data can be enormous. Additionally, Spark needs to hold intermediate results in memory to deduplicate the rows, which could lead to OOM errors if the dataset is too large for the available memory.

How GROUP BY Works in Spark

GROUP BY also involves shuffling, but its behavior can differ:

1. It shuffles data to group by the specified columns.
2. Aggregations like SUM, MAX, etc., are applied during the shuffle process, reducing the amount of data that needs to be shuffled and stored in memory.

Since aggregations reduce the amount of data during the shuffle stage (e.g., combining rows into a single aggregate result for each group), GROUP BY typically uses less memory than DISTINCT.

Why DISTINCT Causes OOM

1. Intermediate Data Volume: Unlike GROUP BY, which reduces data during aggregation, DISTINCT must hold all unique rows in memory until deduplication is complete.
2. Skewed Data: If your data is skewed (some keys or rows appear more frequently), the DISTINCT operation can overload certain partitions, causing uneven memory usage and potential OOM errors.
3. Large Shuffle Size: DISTINCT can create a larger shuffle size compared to GROUP BY, especially when the data contains many duplicates.

Hope this helps!

Mark it as solution if you found it helpful.

Regards,

Avinash N