Appreciate Avinash for the detailed response. I'm using n2 high cpu 16 has 16GB, with 10 workers total 160GB.  Memory is 7616M per executor i.e. executor memory is 77GB in total. Would it not be able to handle the (30GB compressed CSV) data for distinct?  Additionally, why having huge shuffle data could be an issue wrt OOM?