cancel
Showing results for 
Search instead for 
Did you mean: 
Machine Learning
Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithms, model training, deployment, and more. Connect with ML enthusiasts and experts.
cancel
Showing results for 
Search instead for 
Did you mean: 

Working with pyspark dataframe with machine learning libraries / statistical model libraries

javeed
New Contributor

Hi Team, 

I am working with huge volume of data (50GB) and i decompose the time series data using the statsmodel.

Having said that the major challenge i am facing is the compatibility of the pyspark dataframe with the machine learning algorithms. although the pysaprk.ml is available but many of the models are not available.

My question is on a broader level ,How can we handle this compatibility issue ? 

Converting pyspark to pandas has not been proven to be very efficient.

0 REPLIES 0

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group