https://community.databricks.com/t5/data-engineering/how-to-optimize-and-convert-a-spark-dataframe-to-arrow/m-p/26052#M18177 <P>Example use case: When connecting a sample Plotly Dash application to a large dataset, in order to test the performance, I need the file format to be in either hdf5 or arrow.&nbsp;</P><P></P><P>According to this doc:&nbsp;<A href="https://docs.databricks.com/spark/latest/spark-sql/spark-pandas.html" alt="https://docs.databricks.com/spark/latest/spark-sql/spark-pandas.html" target="_blank">Optimize conversion between PySpark and pandas DataFrames</A>, it seems like you can convert between dataframes and Arrow objects by using Pandas as an intermediary,&nbsp;but there are some limitations (e.g. it collects all records in the DataFrame to the driver and should be done on a small subset of the data).</P><P></P><P>How can this be accomplished without type conversion warnings and out-of-memory errors? </P> Mon, 07 Jun 2021 16:51:14 GMT User16776430979 2021-06-07T16:51:14Z https://community.databricks.com/t5/data-engineering/how-to-optimize-and-convert-a-spark-dataframe-to-arrow/m-p/26052#M18177 <P>Example use case: When connecting a sample Plotly Dash application to a large dataset, in order to test the performance, I need the file format to be in either hdf5 or arrow.&nbsp;</P><P></P><P>According to this doc:&nbsp;<A href="https://docs.databricks.com/spark/latest/spark-sql/spark-pandas.html" alt="https://docs.databricks.com/spark/latest/spark-sql/spark-pandas.html" target="_blank">Optimize conversion between PySpark and pandas DataFrames</A>, it seems like you can convert between dataframes and Arrow objects by using Pandas as an intermediary,&nbsp;but there are some limitations (e.g. it collects all records in the DataFrame to the driver and should be done on a small subset of the data).</P><P></P><P>How can this be accomplished without type conversion warnings and out-of-memory errors? </P> Mon, 07 Jun 2021 16:51:14 GMT https://community.databricks.com/t5/data-engineering/how-to-optimize-and-convert-a-spark-dataframe-to-arrow/m-p/26052#M18177 User16776430979 2021-06-07T16:51:14Z