topic Re: Pandas API on Spark, Does it run on a multi-node cluster? in Data Engineering

Pandas API on Spark, Does it run on a multi-node cluster?

Mado — Mon, 17 Oct 2022 22:11:09 GMT

Hi,

I have a few questions about "Pandas API on Spark". Thanks for your time to read my questions

1) Input to these functions are Pandas DataFrame or PySpark DataFrame?

2) When I use any pandas function (like isna, size, apply, where, etc ), does it run only on one node or multi nodes?

Thanks.

Re: Pandas API on Spark, Does it run on a multi-node cluster?

Mado — Tue, 18 Oct 2022 21:21:17 GMT

@Debayan Mukherjee

Thanks for your help.

I have a question about terms : "Pandas dataset" and "pandas-on-Spark dataset".

When you say "dataset", does it refer to "DataFrame"?

If I create "pandas-on-Spark dataset ", can I apply Pandas functions on it, or I should convert it to "pandas dataset" before such a computation?

If I need to convert it to "pandas dataset", I think computation will be done on a single node. Is it correct?

Re: Pandas API on Spark, Does it run on a multi-node cluster?

jose_gonzalez — Mon, 24 Oct 2022 18:49:13 GMT

I would like to share the following information, that might help you.

Pandas API on Spark fills this gap by providing pandas equivalent APIs that work on Apache Spark. Pandas API on Spark is useful not only for pandas users but also PySpark users, because pandas API on Spark supports many tasks that are difficult to do with PySpark, for example plotting data directly from a PySpark DataFrame. Doc https://docs.databricks.com/_static/notebooks/pandas-to-pandas-api-on-spark-in-10-minutes.html

Re: Pandas API on Spark, Does it run on a multi-node cluster?

Mado — Tue, 25 Oct 2022 09:02:02 GMT

Thanks for your reply.

I just want to confirm that Pandas API on Spark uses the parallelism capability of Spark (computations on multi nodes).

Re: Pandas API on Spark, Does it run on a multi-node cluster?

Debayan — Tue, 18 Oct 2022 12:46:39 GMT

Hi @Mohammad Saber ,

Pandas dataset lives in the single machine, and is naturally iterable locally within the same machine. However, pandas-on-Spark dataset lives across multiple machines, and they are computed in a distributed manner. It is difficult to be locally iterable and it is very likely users collect the entire data into the client side without knowing it. Therefore, it is best to stick to using pandas-on-Spark APIs.

Please refer:

https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/best_practices.html#use-pandas-api-on-spark-directly-whenever-possible

https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/index.html

https://docs.databricks.com/languages/pandas-spark.html

Please let us know if you need further clarification on the same. We are more than happy to assist you further.