https://community.databricks.com/t5/data-engineering/what-s-the-difference-between-sparkml-and-spark-mllib/m-p/23234#M15999 <P>They're not really different. Before DataFrames in Spark, older implementations of ML algorithms build on the RDD API. This is generally called "Spark MLlib". After DataFrames, some newer implementations were added as wrappers on top of the old ones that extended the API to work with DataFrames. This is sometimes called "Spark ML". Both are kind of interchangeable; both are still there, and in some cases there are implementations in the 'old' Spark MLlib that aren't also present in the 'newer' Spark ML APIs. So while you probably mostly use the DataFrame API versions, they're kind of one unit, and either term is OK. I think "Spark ML" is more preferred for the newer API, but you'll see MLlib used a lot still.</P> Thu, 17 Jun 2021 18:23:47 GMT sean_owen 2021-06-17T18:23:47Z https://community.databricks.com/t5/data-engineering/what-s-the-difference-between-sparkml-and-spark-mllib/m-p/23233#M15998 <P>I have heard people talk about SparkML but when reading documentation it talks about MLlib. I don't understand the difference, could anyone help me understand this?</P> Thu, 17 Jun 2021 15:02:38 GMT https://community.databricks.com/t5/data-engineering/what-s-the-difference-between-sparkml-and-spark-mllib/m-p/23233#M15998 User16826992666 2021-06-17T15:02:38Z https://community.databricks.com/t5/data-engineering/what-s-the-difference-between-sparkml-and-spark-mllib/m-p/23234#M15999 <P>They're not really different. Before DataFrames in Spark, older implementations of ML algorithms build on the RDD API. This is generally called "Spark MLlib". After DataFrames, some newer implementations were added as wrappers on top of the old ones that extended the API to work with DataFrames. This is sometimes called "Spark ML". Both are kind of interchangeable; both are still there, and in some cases there are implementations in the 'old' Spark MLlib that aren't also present in the 'newer' Spark ML APIs. So while you probably mostly use the DataFrame API versions, they're kind of one unit, and either term is OK. I think "Spark ML" is more preferred for the newer API, but you'll see MLlib used a lot still.</P> Thu, 17 Jun 2021 18:23:47 GMT https://community.databricks.com/t5/data-engineering/what-s-the-difference-between-sparkml-and-spark-mllib/m-p/23234#M15999 sean_owen 2021-06-17T18:23:47Z