https://community.databricks.com/t5/machine-learning/spark-apache-org/m-p/12629#M663 <P>mapInPandas is one of the most powerful Spark functions. It uses an arrow-like in-memory data structure to split up Spark Data Frames into chunks and feeding them to a function that takes a Pandas DF as input and output. Check it out here:</P><P></P><P><A href="https://spark.apache.org/docs/3.0.0/sql-pyspark-pandas-with-arrow.html#map" target="test_blank">https://spark.apache.org/docs/3.0.0/sql-pyspark-pandas-with-arrow.html#map</A></P> Fri, 22 Oct 2021 16:06:35 GMT Dan_Z 2021-10-22T16:06:35Z https://community.databricks.com/t5/machine-learning/spark-apache-org/m-p/12629#M663 <P>mapInPandas is one of the most powerful Spark functions. It uses an arrow-like in-memory data structure to split up Spark Data Frames into chunks and feeding them to a function that takes a Pandas DF as input and output. Check it out here:</P><P></P><P><A href="https://spark.apache.org/docs/3.0.0/sql-pyspark-pandas-with-arrow.html#map" target="test_blank">https://spark.apache.org/docs/3.0.0/sql-pyspark-pandas-with-arrow.html#map</A></P> Fri, 22 Oct 2021 16:06:35 GMT https://community.databricks.com/t5/machine-learning/spark-apache-org/m-p/12629#M663 Dan_Z 2021-10-22T16:06:35Z