topic Re: Do One-Hot-Encoding (OHE) before or after split data to train and test dataframe in Machine Learning

Do One-Hot-Encoding (OHE) before or after split data to train and test dataframe

NhatHoang — Thu, 08 Dec 2022 08:52:29 GMT

Hi,

I wonder that I should do OHE before or after I split data to build up a ML model.

Please give some advise.

Re: Do One-Hot-Encoding (OHE) before or after split data to train and test dataframe

LandanG — Thu, 08 Dec 2022 14:27:47 GMT

Hi @Nhat Hoang ,

While not Databricks-specific, here's a good answer:

"If you perform the encoding before the split, it will lead to data leakage (train-test contamination). In this sense, you will introduce new data (integers of Label Encoders) and use it for your models thus it will affect the end predictions results (good validation scores but poor in deployment).

After the train and validation data category is already matched up, you can perform fit_transform on the train data, then only transform for the validation data - based on the encoding maps from the train data.

Almost all feature engineering like standardization, Normalisation etc should be done after the train test split. "

Additionally, if you were to run an AutoML experiment and look at the underlying notebook you should see that the data is split before encoding is applied.

Re: Do One-Hot-Encoding (OHE) before or after split data to train and test dataframe

NhatHoang — Fri, 09 Dec 2022 02:23:02 GMT

Hi @Landan George ,

Thank you very much. It is clear for me now.

5 stars support, Databricks team. :)

Re: Do One-Hot-Encoding (OHE) before or after split data to train and test dataframe

LandanG — Fri, 09 Dec 2022 14:01:56 GMT

Thank you @Nhat Hoang, I'm glad I could help