Photon and UDF efficiency

Dekova
New Contributor II

When using a JVM engine, Scala UDFs have an advantage over Python UDFs because data doesn't have to be shifted out to the Python environment for processing. If I understand the implications of using the Photon C++ engine, any processing that needs to take place in the JVM (Scala UDFs) or in the Python environment (Python and Pandas UDFs) both need to have data serialized and/or shifted for processing.

Is my understanding correct? Do Scala UDFs lose their performance advantage when using the Photon C++ engine?

-werners-
Esteemed Contributor III

Photon does not support UDFs:
https://learn.microsoft.com/en-us/azure/databricks/runtime/photon#limitations

So when creating a UDF, photon will not be used.

View solution in original post