topic Re: One-hot encoding of strong cardinality features failing, causes downstream issues in Machine Learning

One-hot encoding of strong cardinality features failing, causes downstream issues

rtreves — Fri, 31 May 2024 13:53:35 GMT

Hi Databricks support,

I'm training an ML model using mlflow on DBR 13.3 LTS ML, Spark 3.4.1 using databricks.automl_runtime 0.2.17 and databricks.automl 1.20.3, with shap 0.45.1. My training data has two float-type columns with three or fewer unique values, which automl flags for one-hot encoding. My training experiment finishes without error. When I examined the notebook of the best-performing model, I toggled `shap_enabled` to `True` to see the shap values. However, in the cell that produces shap values, the following error is produced: "TypeError: no supported conversion for types: (dtype('O'),)" (full traceback attached).

From my debugging, I believe the error occurs because the one-hot encoding of the two aforementioned columns fails, leading to object columns being passed to `scipy.sparse.csr_matrix` within the shap package. Indeed, when I go into the training notebook and try to fit the one-hot encoder to the two columns, I get the message "Warning: No categorical columns found. Calling 'transform' will only return input data."

Let me know if a full reprex is needed, and the best way to supply it.

Thanks in advance!

Re: One-hot encoding of strong cardinality features failing, causes downstream issues

NandiniN — Sat, 01 Jun 2024 08:22:57 GMT

Hi @rtreves ,

TypeError: no supported conversion for types: (dtype('O'),)

error means you passed some data type that it doesn’t support, like categorical values (strings probably).

The function expects numeric values, and you provided non-numeric leading to this.

Re: One-hot encoding of strong cardinality features failing, causes downstream issues

rtreves — Sun, 02 Jun 2024 02:50:09 GMT

@NandiniN I've confirmed the features in question are passed in as pandas series with dtype float. Automl flags these features as potentially categorical precisely because they are passed in as numeric types but have few unique values.

Re: One-hot encoding of strong cardinality features failing, causes downstream issues

NandiniN — Sun, 02 Jun 2024 04:57:13 GMT

You earlier mentioned you could share a repro. Can you please do that so that I can check further?

Re: One-hot encoding of strong cardinality features failing, causes downstream issues

rtreves — Mon, 03 Jun 2024 02:38:02 GMT

Hi @NandiniN , thanks for taking a look! I'm linking below a reprex notebook in two formats: .py for running as a databricks notebook on DBR LTS, and a .ipynb notebook for running as a jupyter notebook natively (though I haven't tested this format). Let me know if you have issues getting it running.

https://drive.google.com/drive/folders/1V5hMzGlP3-nxXQUc-g8Y2qhZ3hs40ENs?usp=sharing

Re: One-hot encoding of strong cardinality features failing, causes downstream issues

lilir5 — Mon, 14 Oct 2024 04:19:29 GMT

Hi there,

I have a same issue after running the AutoML without error. Is there any update on this link? Cheers

Re: One-hot encoding of strong cardinality features failing, causes downstream issues

rtreves — Tue, 15 Oct 2024 14:25:03 GMT

No, unfortunately I haven't found any resolution to this issue yet.

Re: One-hot encoding of strong cardinality features failing, causes downstream issues

rtreves — Tue, 15 Oct 2024 14:25:47 GMT

@NandiniN Were you able to use my reprex above to investigate this issue at all? Thank you.

Re: One-hot encoding of strong cardinality features failing, causes downstream issues

NandiniN — Thu, 31 Oct 2024 09:18:39 GMT

Hi @rtreves , sorry I was not able to investigate on the above. Not sure if you would be able to create a support ticket with Databricks as it may be an involved effort to review the code.

I do have a suggestion, instead of relying on the automatic one-hot encoding by AutoML, you can manually perform one-hot encoding on these columns. This way, you can ensure that the encoding is correctly applied and the resulting columns are of the appropriate type.