I think you’re looking at this the right way.

For an exact dedup, there’s no way around a global shuffle somewhere. A separate hash column doesn’t fundamentally change that, since Spark is already hashing the grouping keys internally for distinct().

Since you’ve already ruled out skew, I’d spend more time looking at spill than at the dedup logic itself. In my experience, once skew is gone, the next question is whether the shuffle is memory-bound or simply moving a lot of data.

One thing I’d also validate is where the runtime is actually being spent. I’ve seen cases where people attribute everything to the distinct(), but a noticeable chunk of the wall-clock time is in the write path afterwards, especially when Optimize Write and Auto Compaction are both enabled.

If the UI shows balanced tasks, minimal spill, and no skew, you may already be fairly close to the practical limit for an exact global dedup of this size.