topic Why should I use Spark MLlib for ML vs other available libraries? in Data Engineering https://community.databricks.com/t5/data-engineering/why-should-i-use-spark-mllib-for-ml-vs-other-available-libraries/m-p/24287#M16878 Tue, 15 Jun 2021 20:57:14 GMT User16826992666 2021-06-15T20:57:14Z Why should I use Spark MLlib for ML vs other available libraries? https://community.databricks.com/t5/data-engineering/why-should-i-use-spark-mllib-for-ml-vs-other-available-libraries/m-p/24287#M16878 Tue, 15 Jun 2021 20:57:14 GMT https://community.databricks.com/t5/data-engineering/why-should-i-use-spark-mllib-for-ml-vs-other-available-libraries/m-p/24287#M16878 User16826992666 2021-06-15T20:57:14Z Re: Why should I use Spark MLlib for ML vs other available libraries? https://community.databricks.com/t5/data-engineering/why-should-i-use-spark-mllib-for-ml-vs-other-available-libraries/m-p/24288#M16879 <P>You don't have to. If you don't have a huge data set, there may not be much value in Spark ML over anything else. There are also other distributed modeling libraries that work on Spark like xgboost, and Horovod + TF, Keras, Pytorch. Spark ML is a good choice when you have a very large data set and need a fairly basic algorithm like logistic regression.</P> Thu, 17 Jun 2021 23:08:08 GMT https://community.databricks.com/t5/data-engineering/why-should-i-use-spark-mllib-for-ml-vs-other-available-libraries/m-p/24288#M16879 sean_owen 2021-06-17T23:08:08Z