QuantileDiscretizer is not whiteliested erro!!

[sandeephenkel23]

Dear Team,

We observed that while attempting to use the following import:

from pyspark.sql import functions as F

from pyspark.ml.feature import QuantileDiscretizer

we are encountering the following error:

Py4JSecurityException: QuantileDiscretizer is not whitelisted.

This suggests that the QuantileDiscretizer class is restricted due to security policies within the current environment. Could you kindly advise on how to proceed

[VZLA]

@sandeephenkel23 in which DBR release or environment are you facing this?

[sandeephenkel23]

Hi ,

I am using 13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12)

Can you please help to resolve this?

[VZLA]

@sandeephenkel23 I've run the same code in DBR 13.3 LTS, and 1) It is successfully imported, 2) I can confirm it is in the whitelisted libs. Hence I'm wondering if there's anything else particular in your use case, triggering this? Is your use case a simple DBR 13.3 LTS bare cluster, no additional configs, and also no additional code other than what you have shared here, not used within a UDF, or in a py environment, etc, etc? If you can provide the exact reproducer, we can get to the root cause. Please consider all possible variables, like cluster mode (Assigned, Shared, etc), policies, spark properties, etc.