cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

What's the difference between SparkML and Spark MLlib?

User16826992666
Valued Contributor

I have heard people talk about SparkML but when reading documentation it talks about MLlib. I don't understand the difference, could anyone help me understand this?

1 ACCEPTED SOLUTION

Accepted Solutions

sean_owen
Databricks Employee
Databricks Employee

They're not really different. Before DataFrames in Spark, older implementations of ML algorithms build on the RDD API. This is generally called "Spark MLlib". After DataFrames, some newer implementations were added as wrappers on top of the old ones that extended the API to work with DataFrames. This is sometimes called "Spark ML". Both are kind of interchangeable; both are still there, and in some cases there are implementations in the 'old' Spark MLlib that aren't also present in the 'newer' Spark ML APIs. So while you probably mostly use the DataFrame API versions, they're kind of one unit, and either term is OK. I think "Spark ML" is more preferred for the newer API, but you'll see MLlib used a lot still.

View solution in original post

1 REPLY 1

sean_owen
Databricks Employee
Databricks Employee

They're not really different. Before DataFrames in Spark, older implementations of ML algorithms build on the RDD API. This is generally called "Spark MLlib". After DataFrames, some newer implementations were added as wrappers on top of the old ones that extended the API to work with DataFrames. This is sometimes called "Spark ML". Both are kind of interchangeable; both are still there, and in some cases there are implementations in the 'old' Spark MLlib that aren't also present in the 'newer' Spark ML APIs. So while you probably mostly use the DataFrame API versions, they're kind of one unit, and either term is OK. I think "Spark ML" is more preferred for the newer API, but you'll see MLlib used a lot still.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group