https://community.databricks.com/t5/data-engineering/how-do-i-make-approxsimilarityjoin-work-on-25k-300-d-vectors/m-p/16862#M10959 <P></P> <P>Use a PandasUDF with Arrow enabled. They are improved in Spark 3, but you can use them in Spark 2.4.5.</P> <P></P> Fri, 06 Aug 2021 23:13:06 GMT Dan_Z 2021-08-06T23:13:06Z https://community.databricks.com/t5/data-engineering/how-do-i-make-approxsimilarityjoin-work-on-25k-300-d-vectors/m-p/16861#M10958 <P></P> <P>I’m trying to use LSH approxSimilarityJoin on a dataset with ~25k 300-d vectors of floats. It gets stuck and eventually fails with ’Slave lost’ error. The size of cluster and memory are likely not a problem, the failure happens even with 16 nodes, 16 cores each, 64G RAM (driver of the same size). What would be your suggestions - how do I make it work?</P> <P>Using Spark 2.4.5 on GCP DataProc</P> <P></P> Thu, 05 Aug 2021 23:47:47 GMT https://community.databricks.com/t5/data-engineering/how-do-i-make-approxsimilarityjoin-work-on-25k-300-d-vectors/m-p/16861#M10958 maffeenAF 2021-08-05T23:47:47Z https://community.databricks.com/t5/data-engineering/how-do-i-make-approxsimilarityjoin-work-on-25k-300-d-vectors/m-p/16862#M10959 <P></P> <P>Use a PandasUDF with Arrow enabled. They are improved in Spark 3, but you can use them in Spark 2.4.5.</P> <P></P> Fri, 06 Aug 2021 23:13:06 GMT https://community.databricks.com/t5/data-engineering/how-do-i-make-approxsimilarityjoin-work-on-25k-300-d-vectors/m-p/16862#M10959 Dan_Z 2021-08-06T23:13:06Z