https://community.databricks.com/t5/data-engineering/how-to-optimize-conversion-between-pyspark-and-arrow/m-p/26179#M18292 <P>Seems like you can convert between dataframes and Arrow objects by using Pandas as an intermediary,&nbsp;but there are some limitations (e.g. it collects all records in the DataFrame to the driver and should be done on a small subset of the data, you hit type conversion warnings and run out of memory).</P><P></P><P>What's a more efficient and optimized way to convert from PySpark to Arrow? </P> Fri, 04 Jun 2021 22:57:27 GMT User16776430979 2021-06-04T22:57:27Z https://community.databricks.com/t5/data-engineering/how-to-optimize-conversion-between-pyspark-and-arrow/m-p/26179#M18292 <P>Seems like you can convert between dataframes and Arrow objects by using Pandas as an intermediary,&nbsp;but there are some limitations (e.g. it collects all records in the DataFrame to the driver and should be done on a small subset of the data, you hit type conversion warnings and run out of memory).</P><P></P><P>What's a more efficient and optimized way to convert from PySpark to Arrow? </P> Fri, 04 Jun 2021 22:57:27 GMT https://community.databricks.com/t5/data-engineering/how-to-optimize-conversion-between-pyspark-and-arrow/m-p/26179#M18292 User16776430979 2021-06-04T22:57:27Z