03-13-2023 10:45 AM
Hi,
I have UDF which runs for each spark dataframe row, does some complex processing and return string output. But it takes very long if data is 15000 rows. I have configured cluster with autoscaling, but its not spinning more servers.
Please suggest how to make UDF fasters or any reference implementations.
Regards,
Sanjay
03-14-2023 05:00 AM
Hi @Sanjay Jain , Python UDFs are generally slower to process because it runs mostly in the driver which can also lead to OOM errors on Driver. To resolve this issue, please consider the below:
03-18-2023 03:38 AM
Hi @Sanjay Jain , We haven't heard from you since the last response from @Lakshay Goel, @rishabh and @Vigneshraja Palaniraj, and I was checking back to see if their suggestions helped you.
Or else, If you have any solution, please share it with the community, as it can be helpful to others.
Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.
03-13-2023 11:22 AM
@Sanjay Jain Hi Sanjay. You did not mention what kind of processing you are doing in UDF. Python UDF definitely will create performance issues as Spark optimizer does not apply optimization on what you are doing within the UDF. Please see if you can do any of those processing using Spark native functions.
If still, you need to use python UDF, see if you can try with Pandas UDF. This can provide significant performance improvements for certain types of operations. Pandas UDFs use Apache Arrow to transfer data between Python and Spark, which can result in faster processing times.
03-13-2023 01:22 PM
Write ...whether you can perform the same things by using pyspark native logics and functions then no need to use a UDF. Because in most cases we can do by using pyspark also because UDF will definitely create a performance issues
03-14-2023 05:00 AM
Hi @Sanjay Jain , Python UDFs are generally slower to process because it runs mostly in the driver which can also lead to OOM errors on Driver. To resolve this issue, please consider the below:
03-18-2023 03:38 AM
Hi @Sanjay Jain , We haven't heard from you since the last response from @Lakshay Goel, @rishabh and @Vigneshraja Palaniraj, and I was checking back to see if their suggestions helped you.
Or else, If you have any solution, please share it with the community, as it can be helpful to others.
Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.
Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections.
Click here to register and join today!
Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.