Orianh
Valued Contributor II

@Cheuk Hin Christophe Poon​ 

Well, I don't have any decent idea besides check why this happening, Maybe like you said - the transpose require to collect all the data into driver, maybe you can find valuable information in the spark UI.

Hope you will solve this problem, but its really sound weird that you need 128GB machine to transpose 2 GB matrix.