Example use case: When connecting a sample Plotly Dash application to a large dataset, in order to test the performance, I need the file format to be in either hdf5 or arrow.
According to this doc: Optimize conversion between PySpark and pandas DataFrames, it seems like you can convert between dataframes and Arrow objects by using Pandas as an intermediary, but there are some limitations (e.g. it collects all records in the DataFrame to the driver and should be done on a small subset of the data).
How can this be accomplished without type conversion warnings and out-of-memory errors?
