cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

What's the difference between SparkML and Spark MLlib?

User16826992666
Valued Contributor

I have heard people talk about SparkML but when reading documentation it talks about MLlib. I don't understand the difference, could anyone help me understand this?

1 ACCEPTED SOLUTION

Accepted Solutions

sean_owen
Honored Contributor II
Honored Contributor II

They're not really different. Before DataFrames in Spark, older implementations of ML algorithms build on the RDD API. This is generally called "Spark MLlib". After DataFrames, some newer implementations were added as wrappers on top of the old ones that extended the API to work with DataFrames. This is sometimes called "Spark ML". Both are kind of interchangeable; both are still there, and in some cases there are implementations in the 'old' Spark MLlib that aren't also present in the 'newer' Spark ML APIs. So while you probably mostly use the DataFrame API versions, they're kind of one unit, and either term is OK. I think "Spark ML" is more preferred for the newer API, but you'll see MLlib used a lot still.

View solution in original post

1 REPLY 1

sean_owen
Honored Contributor II
Honored Contributor II

They're not really different. Before DataFrames in Spark, older implementations of ML algorithms build on the RDD API. This is generally called "Spark MLlib". After DataFrames, some newer implementations were added as wrappers on top of the old ones that extended the API to work with DataFrames. This is sometimes called "Spark ML". Both are kind of interchangeable; both are still there, and in some cases there are implementations in the 'old' Spark MLlib that aren't also present in the 'newer' Spark ML APIs. So while you probably mostly use the DataFrame API versions, they're kind of one unit, and either term is OK. I think "Spark ML" is more preferred for the newer API, but you'll see MLlib used a lot still.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.