Possible Incorrect Answer Key: Spark ML Estimators — OneHotEncoder, Imputer, and StandardScaler

[amittian]

Hello Databricks Community,

While preparing for the Databricks Machine Learning Associate certification, I came across a question in a practice assessment that I believe may have an incorrect or incomplete answer key.

Question:

"In a Spark ML Pipeline, which two of the following are Estimators — meaning they learn parameters from the training data during fit()? (Select two)"

Options:

1. OneHotEncoder

2. VectorAssembler

3. Imputer

4. StandardScaler

My selected answers:

• OneHotEncoder

• StandardScaler

Assessment result: Partially correct — 2.5 out of 5 marks.

Answers provided by the assessment:

• Imputer

• StandardScaler

My understanding:

According to the Apache Spark ML API, three of the listed components are Estimators:

• OneHotEncoder: An Estimator that learns category sizes during fit() and produces a OneHotEncoderModel.

• Imputer: An Estimator that learns imputation statistics and produces an ImputerModel.

• StandardScaler: An Estimator that learns feature statistics and produces a StandardScalerModel.

VectorAssembler is a Transformer because it combines feature columns without learning parameters through fit().

Therefore, technically, the question appears to have three correct answers rather than two.

Could someone from the Databricks training or certification team clarify whether the question or its answer key needs correction?

I would appreciate clarification so that learners can align their understanding with the actual Spark ML API.

Thank you!

[NinaThoni]

Hi @amittian,

Thanks for flagging this, you're right.
Since Spark 3.0, OneHotEncoder is an Estimator. It learns the category sizes during fit() and returns a OneHotEncoderModel, so three of the four options are Estimators.

The likely cause is that the question was written for Spark 2.x. In that version, OneHotEncoder was a Transformer and the learning version was a separate class called OneHotEncoderEstimator. Spark 3.0 merged the two under the OneHotEncoder name, and the answer key doesn't seem to have been updated.

I checked with our certification team. They asked for a ticket here so the question can be formally reviewed: https://help.databricks.com/s/contact-us?ReqType=training

If you'd rather I file it for you, just send me a bit more context, such as which exam or assessment it was and roughly when you took it, and I'll take care of it.

Thanks again for raising it!
Nina