So, based on my first recommendation, here is a sample pattern.

from pyspark.sql import functions as F

cols = required_cols

dedup_df = (
    df.select(*cols)
      .withColumn(
          "dedup_key",
          F.sha2(F.to_json(F.struct(*[F.col(c) for c in cols])), 256)
      )
      .repartition("dedup_key")
      .dropDuplicates(["dedup_key"])
      .drop("dedup_key")
)

dedup_df.write.mode("overwrite").saveAsTable("target_table")

The reason I prefer this over raw distinct() is not that it eliminates the shuffle completely, because it does not. The benefit is that Spark now shuffles and compares a narrow synthetic key instead of carrying the full 20-column row through the dedup logic, and that same key can later be reused if you move to incremental dedup. However, I also note your comment that you have already tried this and have not seen any improvement. Is my understanding correct? 

Since you have already checked the Spark UI and confirmed there is no meaningful skew, and since you have already tested different partition counts and found 400 to be the best setting for this load, I would not keep pushing on shuffle tuning. That tells us the bottleneck is not poor distribution or under-partitioning anymore. At that point, the remaining cost is the exact deduplication itself, which is expected for a global DISTINCT over 100M rows.

I would still check the physical plan once with explain("formatted"), but only for one specific reason...to confirm whether Spark is using a HashAggregate-based dedup path or falling back to SortAggregate. If it is SortAggregate, that can make the query materially more expensive because Spark has to sort within partitions before aggregating. If it is already using HashAggregate, then that confirms the main issue is simply the unavoidable cost of exact global dedup, not a bad execution strategy.