- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-17-2021 08:02 AM
I have heard people talk about SparkML but when reading documentation it talks about MLlib. I don't understand the difference, could anyone help me understand this?
- Labels:
-
ML
-
Spark MLlib
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-17-2021 11:23 AM
They're not really different. Before DataFrames in Spark, older implementations of ML algorithms build on the RDD API. This is generally called "Spark MLlib". After DataFrames, some newer implementations were added as wrappers on top of the old ones that extended the API to work with DataFrames. This is sometimes called "Spark ML". Both are kind of interchangeable; both are still there, and in some cases there are implementations in the 'old' Spark MLlib that aren't also present in the 'newer' Spark ML APIs. So while you probably mostly use the DataFrame API versions, they're kind of one unit, and either term is OK. I think "Spark ML" is more preferred for the newer API, but you'll see MLlib used a lot still.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-17-2021 11:23 AM
They're not really different. Before DataFrames in Spark, older implementations of ML algorithms build on the RDD API. This is generally called "Spark MLlib". After DataFrames, some newer implementations were added as wrappers on top of the old ones that extended the API to work with DataFrames. This is sometimes called "Spark ML". Both are kind of interchangeable; both are still there, and in some cases there are implementations in the 'old' Spark MLlib that aren't also present in the 'newer' Spark ML APIs. So while you probably mostly use the DataFrame API versions, they're kind of one unit, and either term is OK. I think "Spark ML" is more preferred for the newer API, but you'll see MLlib used a lot still.
![](/skins/images/B38AF44D4BD6CE643D2A527BE673CCF6/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/B38AF44D4BD6CE643D2A527BE673CCF6/responsive_peak/images/icon_anonymous_message.png)