Why should I use Spark MLlib for ML vs other available libraries?

[sean_owen · Accepted solution]

You don't have to. If you don't have a huge data set, there may not be much value in Spark ML over anything else. There are also other distributed modeling libraries that work on Spark like xgboost, and Horovod + TF, Keras, Pytorch. Spark ML is a good choice when you have a very large data set and need a fairly basic algorithm like logistic regression.