That’s useful context.

If skew is already ruled out and the write phase is relatively fast, I’d focus on two Spark UI metrics next: shuffle write volume and spill (memory/disk). Those will tell you whether there’s still a tuning opportunity or whether you’re simply paying the unavoidable cost of a large global dedup.

One thing I’d be curious about: do duplicates tend to arrive close together in the source data (for example, log retransmits or replay scenarios)? If so, improving duplicate locality before the shuffle can sometimes reduce the amount of data that survives into the global dedup stage.