I’m trying to use LSH approxSimilarityJoin on a dataset with ~25k 300-d vectors of floats. It gets stuck and eventually fails with ’Slave lost’ error. The size of cluster and memory are likely not a problem, the failure happens even with 16 nodes, 16 cores each, 64G RAM (driver of the same size). What would be your suggestions - how do I make it work?
Using Spark 2.4.5 on GCP DataProc
