Hi, 
You can read about Bloom Filter. It can drastically decrease the I/O and also have small footprint.
It tells the spark, that the join id is not definitely in a partition (No False Negative , 100% correct)
Bloom filters only work for equality (=) joins.

I would recommend trying Liquid Clustering and Bloom filter combination.
Let me know if this improves your big data movement issues.