https://community.databricks.com/t5/get-started-discussions/photon-and-udf-efficiency/m-p/38570#M5533 <P>When using a JVM engine, Scala UDFs have an advantage over Python UDFs because data doesn't have to be shifted out to the Python environment for processing. If I understand the implications of using the Photon C++ engine, any processing that needs to take place in the JVM (Scala UDFs) or in the Python environment (Python and Pandas UDFs) both need to have data serialized and/or shifted for processing.</P><P>Is my understanding correct? Do Scala UDFs lose their performance advantage when using the Photon C++ engine?</P> Thu, 27 Jul 2023 12:05:30 GMT Dekova 2023-07-27T12:05:30Z https://community.databricks.com/t5/get-started-discussions/photon-and-udf-efficiency/m-p/38570#M5533 <P>When using a JVM engine, Scala UDFs have an advantage over Python UDFs because data doesn't have to be shifted out to the Python environment for processing. If I understand the implications of using the Photon C++ engine, any processing that needs to take place in the JVM (Scala UDFs) or in the Python environment (Python and Pandas UDFs) both need to have data serialized and/or shifted for processing.</P><P>Is my understanding correct? Do Scala UDFs lose their performance advantage when using the Photon C++ engine?</P> Thu, 27 Jul 2023 12:05:30 GMT https://community.databricks.com/t5/get-started-discussions/photon-and-udf-efficiency/m-p/38570#M5533 Dekova 2023-07-27T12:05:30Z https://community.databricks.com/t5/get-started-discussions/photon-and-udf-efficiency/m-p/38580#M5534 <P>Photon does not support UDFs:<BR /><A href="https://learn.microsoft.com/en-us/azure/databricks/runtime/photon#limitations" target="_self">https://learn.microsoft.com/en-us/azure/databricks/runtime/photon#limitations</A></P><P>So when creating a UDF, photon will not be used.</P> Thu, 27 Jul 2023 14:01:15 GMT https://community.databricks.com/t5/get-started-discussions/photon-and-udf-efficiency/m-p/38580#M5534 -werners- 2023-07-27T14:01:15Z