topic Re: How to apply Pandas functions on PySpark DataFrame? in Data Engineering

How to apply Pandas functions on PySpark DataFrame?

Mado — Sat, 22 Oct 2022 10:38:00 GMT

Hi,

I want to apply Pandas functions (like isna, concat, append, etc) on PySpark DataFrame in such a way that computations are done on multi-node cluster.

I don't want to convert PySpark DataFrame into Pandas DataFrame since, I think, only one node is used for computation.

What is the best way you suggest to use Pandas functions on PySpark DataFrame while having all processes on multi-node cluster?

Re: How to apply Pandas functions on PySpark DataFrame?

Hubert-Dudek — Sun, 23 Oct 2022 21:00:08 GMT

The best is to use pandas on a spark, it is virtually interchangeable so it just different API for Spark data frame

import pyspark.pandas as ps
 
psdf = ps.range(10)
sdf = psdf.to_spark().filter("id > 5")
sdf.show()

Re: How to apply Pandas functions on PySpark DataFrame?

Mado — Sun, 23 Oct 2022 22:06:06 GMT

Thanks for your reply.

I want to apply Pandas function on PySpark DataFrame (like how I use Pandas on DataFrames on a local laptop). But, I think the above example uses PySpark function "filter".