Need some advice from the community.

I am processing around 100 million records using:

df.select(required_cols).distinct().write.saveAsTable(...)

The source has 1000+ columns, but I'm selecting only 20 columns before applying DISTINCT.

I have already enabled:

• AQE
• Photon
• Optimize Write
• Auto Compaction
• Shuffle Partition to 400
• coalescePartitions

DISTINCT is still the biggest bottleneck due to the shuffle.

For log data that cannot rely on any primary key or composite key, what is the best approach for deduplication at scale?
Worker Type is Standard_DS15_v2, and min:5 to max:7 workers is the current configuration.