I think a hash based approach is worth trying here. Since you are deduping on 20 columns, distinct() has to shuffle and compare all those columns, which can become expensive at this scale.

With a hash column, Spark still has to shuffle, but the comparison becomes lighter because it is working with one deterministic hash value instead of all 20 columns. So it may not remove the bottleneck completely, but it can reduce the shuffle/comparison cost.

There will be some extra CPU cost to generate the hash, so I would still benchmark both approaches. For recurring log dedup, I would lean toward generating a hash from the selected fields and using it as a technical dedup key, especially if the process can be made incremental instead of deduping the full dataset every run.