Hi,

I have UDF which runs for each spark dataframe row, does some complex processing and return string output. But it takes very long if data is 15000 rows. I have configured cluster with autoscaling, but its not spinning more servers.

Please suggest how to make UDF fasters or any reference implementations.

Regards,

Sanjay