Pyspark models iterative/augmented training capability
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-12-2024 12:01 PM
Does Pyspark tree based models have iterative or augmented training capabilities ? Similar to sklearn package can be used to train models using model artifact and use that model to train using additional data?
#ML_Models_Pyspark
1 REPLY 1
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-05-2024 12:36 PM
Hi @ChanduBhujang,
Thank you for contacting Databricks community.
PySpark tree-based models do not have built-in iterative or augmented training capabilities like Scikit-learn's
partial_fit
method. While there are workarounds to update the model with new data, they may not be as efficient or effective as native support for incremental training.
