Python UDF vs Scala UDF in pyspark code

greyamber
New Contributor II

Is there a performance difference between Python UDF vs Scala UDF in pyspark code.

szymon_dybczak
Esteemed Contributor III

Hi @greyamber ,

Yes, there is a difference. Scala would be faster. You read about the reason and benchmark on following blog:

Spark UDF — Deep Insights in Performance | by QuantumBlack, AI by McKinsey | QuantumBlack, AI by McK...