https://community.databricks.com/t5/data-engineering/python-udf-vs-scala-udf-in-pyspark-code/m-p/80995#M36191 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/96108">@greyamber</a>&nbsp;,</P><P>Yes, there is a difference. Scala would be faster. You read about the reason and benchmark on following blog:<BR /><BR /><A href="https://medium.com/quantumblack/spark-udf-deep-insights-in-performance-f0a95a4d8c62" target="_blank">Spark UDF — Deep Insights in Performance | by QuantumBlack, AI by McKinsey | QuantumBlack, AI by McKinsey | Medium</A></P> Mon, 29 Jul 2024 15:48:33 GMT szymon_dybczak 2024-07-29T15:48:33Z https://community.databricks.com/t5/data-engineering/python-udf-vs-scala-udf-in-pyspark-code/m-p/80992#M36190 <P>Is there a performance difference between Python UDF vs Scala UDF in pyspark code.</P> Mon, 29 Jul 2024 15:42:17 GMT https://community.databricks.com/t5/data-engineering/python-udf-vs-scala-udf-in-pyspark-code/m-p/80992#M36190 greyamber 2024-07-29T15:42:17Z https://community.databricks.com/t5/data-engineering/python-udf-vs-scala-udf-in-pyspark-code/m-p/80995#M36191 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/96108">@greyamber</a>&nbsp;,</P><P>Yes, there is a difference. Scala would be faster. You read about the reason and benchmark on following blog:<BR /><BR /><A href="https://medium.com/quantumblack/spark-udf-deep-insights-in-performance-f0a95a4d8c62" target="_blank">Spark UDF — Deep Insights in Performance | by QuantumBlack, AI by McKinsey | QuantumBlack, AI by McKinsey | Medium</A></P> Mon, 29 Jul 2024 15:48:33 GMT https://community.databricks.com/t5/data-engineering/python-udf-vs-scala-udf-in-pyspark-code/m-p/80995#M36191 szymon_dybczak 2024-07-29T15:48:33Z