topic Re: Why do Spark MLlib models only accept a vector column as input? in Data Engineering

Why do Spark MLlib models only accept a vector column as input?

User16826992666 — Tue, 15 Jun 2021 21:10:04 GMT

In other libraries I can just use the feature columns themselves as inputs, why do I need to make a vector out of my features when I use MLlib?

Re: Why do Spark MLlib models only accept a vector column as input?

User16826992666 — Wed, 16 Jun 2021 22:14:51 GMT

The modeling algorithms in Spark MLlib will only accept a vectorized column as input. This is done for reasons of efficiency and scaling.

The vector assembler will express the features efficiently using techniques like spark vector, which allow a larger amount of data to be handled with less memory. This helps the modeling algorithms run efficiently even on large data columns.

Re: Why do Spark MLlib models only accept a vector column as input?

sean_owen — Thu, 17 Jun 2021 23:05:12 GMT

Yeah, it's more a design choice. Rather than have every implementation take column(s) params, this is handled once in VectorAssembler for all of them. One way or the other, most implementations need a vector of inputs anyway. VectorAssembler can do some optimizations to use sparse vectors too where applicable.