How do I make approxSimilarityJoin work on 25k 300-d vectors?

maffeenAF
New Contributor

‎08-05-2021 04:47 PM

I’m trying to use LSH approxSimilarityJoin on a dataset with ~25k 300-d vectors of floats. It gets stuck and eventually fails with ’Slave lost’ error. The size of cluster and memory are likely not a problem, the failure happens even with 16 nodes, 16 cores each, 64G RAM (driver of the same size). What would be your suggestions - how do I make it work?

Using Spark 2.4.5 on GCP DataProc

0 Kudos

Dan_Z
Databricks Employee
Databricks Employee

‎08-06-2021 04:13 PM

Use a PandasUDF with Arrow enabled. They are improved in Spark 3, but you can use them in Spark 2.4.5.

0 Kudos