topic Re: pandas udf type grouped map fails in Data Engineering

pandas udf type grouped map fails

user_b22ce5eeAl — Fri, 13 Aug 2021 14:07:18 GMT

Hello,

I am trying to get the shap values for my whole dataset using pandas udf for each category of a categorical variable. It runs well when I run it on a few categories but when I want to run the function on the whole dataset my job fails. I see spills both on memory and disk and my shuffle read is around 40GB. I am not sure how to optimize my spark job here, I increased my cores to 160 and also Memory for both driver and workers but still not successful.

Any suggestion will be highly appreciated.

Thanks

Re: pandas udf type grouped map fails

user_b22ce5eeAl — Mon, 16 Aug 2021 14:23:13 GMT

was able to get it done by increasing driver memory!

Re: pandas udf type grouped map fails

Jackson — Tue, 17 Aug 2021 04:01:03 GMT

I want to use data.groupby.apply() to apply a function to each row of my Pyspark Dataframe per group.

I used The Grouped Map Pandas UDFs. However I can't figure out how to add another argument to my function. DGCustomerFirst Survey

I tried using the argument as a global variable but the function doesn't recongnize it ( my argument is a pyspark dataframe)