Pyspark models iterative/augmented training capability

ChanduBhujang
New Contributor II

Does Pyspark tree based models have iterative or augmented training capabilities ? Similar to sklearn package can be used to train models using model artifact and use that model to train using additional data? 

 

#ML_Models_Pyspark

Kumaran
Databricks Employee
Databricks Employee

Hi @ChanduBhujang,

Thank you for contacting Databricks community.

PySpark tree-based models do not have built-in iterative or augmented training capabilities like Scikit-learn's partial_fit method. While there are workarounds to update the model with new data, they may not be as efficient or effective as native support for incremental training.