Fuzzy Match on PySpark using UDF/Pandas UDF

mohaimen_syed
New Contributor III

‎10-18-2023 03:20 PM

I'm trying to do fuzzy matching on two dataframes by cross joining them and then using a udf for my fuzzy matching. But using both python udf and pandas udf its either very slow or I get an error.

 

@pandas_udf("int")
def core_match_processor(s1: pd.Series, s2: pd.Series) -> pd.Series:
return pd.Series(int(rapidfuzz.ratio(s1, s2)))

MatchUDF = f.pandas_udf(core_match_processor, returnType=IntegerType())
 
df0 = df1.crossJoin(broadcast(df2))
df = df0.withColumn("Score", MatchUDF(f.col("String1"), f.col("String2")))
 
Error: org.apache.spark.SparkRuntimeException: [UDF_USER_CODE_ERROR.GENERIC] Execution of function core_match_processor
0 Kudos

mohaimen_syed
New Contributor III

‎10-19-2023 03:28 PM

I'm now getting the error: (SQL_GROUPED_AGG_PANDAS_UDF) is not supported on clusters in Shared access mode.
Even though this article clearly states that pandas udf is supported for shared cluster in databricks

https://www.databricks.com/blog/shared-clusters-unity-catalog-win-introducing-cluster-libraries-pyth...

0 Kudos

mohaimen_syed
New Contributor III
In response to Retired_mod

‎10-20-2023 11:13 AM

Cluster:
Policy: Shared Compute

Access: Shared

Runtime: 14.1 (includes Apache Spark 3.5.0, Scala 2.12)

Worker type: Standard_L8s_v3 (64 GB Memory, 8 Cores) - workers- 1-60
Driver type: Standard_L8s_v3 (64 GB Memory, 8 Cores)

I added this line in my python notebook: 

spark.conf.set("spark.sql.execution.arrow.pyspark.enabled", "true") which I believe will enable Apache Apache Arrow optimization.

thibault
Contributor III
In response to mohaimen_syed

‎03-26-2024 03:52 AM

Any updates here? I'm running into the same problem with serverless compute

0 Kudos