Hello,
I recently discovered the sparklyr package and found it quite useful. After setting up the Spark connection, I can apply dplyr functions to manipulate large tables. However, it seems that any functions outside of dplyr cannot be used on Spark views. For instance, after manipulating the data, I wanted to transpose it using pivot_wider, but this function is only available in the tidyr package. Does anyone know of any R packages that would allow me to use most of the tidyverse syntax on Spark dataframes to process large dataframes quickly?
Thanks.
