Working with pyspark dataframe with machine learning libraries / statistical model libraries

Machine Learning

Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.

Hi Team,

I am working with huge volume of data (50GB) and i decompose the time series data using the statsmodel.

Having said that the major challenge i am facing is the compatibility of the pyspark dataframe with the machine learning algorithms. although the pysaprk.ml is available but many of the models are not available.

My question is on a broader level ,How can we handle this compatibility issue ?

Converting pyspark to pandas has not been proven to be very efficient.