topic Re: Fuzzy text matching in Spark in Data Engineering

Fuzzy text matching in Spark

manugarri — Tue, 15 Mar 2016 11:09:04 GMT

I have a list of client provided data, a list of company names.

I have to match those names with an internal database of company names. The client list can fit in memory (its about 10k elements) but the internal dataset is on hdfs and we use Spark for accesing it.

How could I go and match the client list? I was thinking in doing a matrix (RowMatrix) of N x D elements, n being the number of client elements and D being the length of the internal client list) and compute the similarities pair wise.

How could I do this in Spark? Any help would be more than welcome.

Re: Fuzzy text matching in Spark

vida — Fri, 01 Apr 2016 17:04:29 GMT

You can use python libraries in Spark. I suggest using fuzzy-wuzzy for computing the similarities.

Then you just need to join the client list with the internal dataset. If you wanted to make sure you tried every single client list against the internal dataset, then you can do a cartesian join. But there may be a better way to cut down the possibilities so you can use a more efficient join - such as assuming the internal dataset name starts with the same letter as the client list. You can even try multiple passes on the internal dataset and try more complicated logic each time.

Re: Fuzzy text matching in Spark

Bill_Chambers — Fri, 01 Apr 2016 17:05:51 GMT

I'm not aware of any solution out of the box to be able to do something like this but there are several talks that have been done on the subject which you can find below.

https://spark-summit.org/2015/events/real-time-fuzzy-matching-with-spark-and-elastic-search/

https://spark-summit.org/2014/talk/fuzzy-matching-with-spark

Re: Fuzzy text matching in Spark

manugarri — Fri, 01 Apr 2016 17:37:36 GMT

Yeah, those two examples (which are the top ones that appear on google) reference a talk which basically doesnt explain how to implement anything.

Re: Fuzzy text matching in Spark

PaulExter — Wed, 09 Aug 2017 21:14:27 GMT

Curious if you ever found a workable solution to this. Your question is still one of the top hits when I Google it. We are facing a similar challenge, where we want to be able to fuzzy match high volume lists of individuals in HDFS / Hive. Thinking of creating something in PySpark, or implementing Elastic, but don't want to reinvent the wheel if there's something already out there. We need to standardize our data before matching as well, but that's another story.

Re: Fuzzy text matching in Spark

MatiasRotenberg — Thu, 10 Aug 2017 07:41:49 GMT

Like vida said, you can use python libraries to get text matching algorithms.

You can even register the function and use it as a udf in SQL.

Re: Fuzzy text matching in Spark

manugarri — Thu, 10 Aug 2017 08:26:36 GMT

Matias, in my experience using python udfs is tremendously slow.

Re: Fuzzy text matching in Spark

hansonkx — Wed, 29 Nov 2017 19:01:22 GMT

for those of you looking for a not very complicated solution, you can use the 2 native spark api Soundex and Levenshtein as your fuzzy matching algorithms.

val joinedDF = accountDF.join( accountDF2, levenshtein(accountDF("name"), accountDF2("name")) < 3 && (accountDF("id") !== accountDF2("id")) )

joinedDF.show

Re: Fuzzy text matching in Spark

hansonkx — Wed, 29 Nov 2017 19:05:13 GMT

for those of you who are looking for a not too complicated solution, you can use the two built in spark api soundex and levenshtein

val newDF = accountDF.join(
  accountDF2,
  levenshtein(accountDF("name"), accountDF2("name")) < 3 && (accountDF("id") !== accountDF2("id"))
)
newDF.show

Re: Fuzzy text matching in Spark

Er__Ram_Saran_B — Wed, 05 Jun 2019 03:47:36 GMT

The great question about Fuzzy text matching in Spark, this is unique topic, and part of fuzzy Logic .

Thanks

Re: Fuzzy text matching in Spark

Sonal — Tue, 14 Sep 2021 07:13:20 GMT

You can use Zingg: Spark based open source tool for this https://github.com/zinggAI/zingg

Re: Fuzzy text matching in Spark

Edthehead — Thu, 16 Oct 2025 06:16:55 GMT

You can refer to this article Optimizing Large-Scale Fuzzy Matching with Apache Spark and Databricks | by Gavaragirijarani | Medium.

As far as open-source libraries go, rapidfuzz is known to be faster than fuzzywuzzy.

Re: Fuzzy text matching in Spark

Shamzaa3Q — Mon, 20 Oct 2025 04:28:57 GMT

+1 for rapidfuzz, I have used it in production pipelines. Better than just levenshtein function, as rapidfuzz provides a couple of other algorithms as well. I will warn you to not do what 2024 me attempted, which is use LLM to solve for this. It sounded like a good use-case until I implemented it and the $$ spend to value returned was terrible.

Re: Fuzzy text matching in Spark

RheaC — Thu, 05 Mar 2026 07:23:52 GMT

+1 on LLMs. I would check this article on using Similarity API instead of rapidfuzz in 2026 especially for larger/growing datasets https://medium.com/p/0854593e380a