topic Re: QuantileDiscretizer is not whiteliested erro!! in Data Engineering

QuantileDiscretizer is not whiteliested erro!!

sandeephenkel23 — Fri, 25 Oct 2024 05:30:32 GMT

Dear Team,

We observed that while attempting to use the following import:

from pyspark.sql import functions as F

from pyspark.ml.feature import QuantileDiscretizer

we are encountering the following error:

Py4JSecurityException: QuantileDiscretizer is not whitelisted.

This suggests that the QuantileDiscretizer class is restricted due to security policies within the current environment. Could you kindly advise on how to proceed

Re: QuantileDiscretizer is not whiteliested erro!!

VZLA — Thu, 31 Oct 2024 13:41:31 GMT

@sandeephenkel23 in which DBR release or environment are you facing this?

Re: QuantileDiscretizer is not whiteliested erro!!

sandeephenkel23 — Mon, 04 Nov 2024 11:45:42 GMT

Hi ,

I am using 13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12)

Can you please help to resolve this?

Re: QuantileDiscretizer is not whiteliested erro!!

VZLA — Mon, 04 Nov 2024 12:05:50 GMT

@sandeephenkel23 I've run the same code in DBR 13.3 LTS, and 1) It is successfully imported, 2) I can confirm it is in the whitelisted libs. Hence I'm wondering if there's anything else particular in your use case, triggering this? Is your use case a simple DBR 13.3 LTS bare cluster, no additional configs, and also no additional code other than what you have shared here, not used within a UDF, or in a py environment, etc, etc? If you can provide the exact reproducer, we can get to the root cause. Please consider all possible variables, like cluster mode (Assigned, Shared, etc), policies, spark properties, etc.