cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

sanjay
by Valued Contributor II
  • 5148 Views
  • 4 replies
  • 4 kudos

Resolved! PySpark UDF is taking long to process

Hi,I have UDF which runs for each spark dataframe row, does some complex processing and return string output. But it takes very long if data is 15000 rows. I have configured cluster with autoscaling, but its not spinning more servers.Please suggest h...

  • 5148 Views
  • 4 replies
  • 4 kudos
Latest Reply
Kaniz
Community Manager
  • 4 kudos

Hi @Sanjay Jain​ ​​, We haven't heard from you since the last response from @Lakshay Goel​, @rishabh and @Vigneshraja Palaniraj​​, and I was checking back to see if their suggestions helped you.Or else, If you have any solution, please share it with ...

  • 4 kudos
3 More Replies
Labels