Next, use the repartition(n) to increase your dataframe to twice the number you got earlier. For example, if num_partitions was 30, then repartition(60) prior to running your query.  With half the data in each Memory Partition, I'm guessing you won't OOM.  If you still do, increase the number by x2 again until the OOM disappears..