QuantileDiscretizer is not whiteliested erro!!
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-24-2024 10:30 PM
Dear Team,
We observed that while attempting to use the following import:
from pyspark.sql import functions as F
from pyspark.ml.feature import QuantileDiscretizer
we are encountering the following error:
Py4JSecurityException: QuantileDiscretizer is not whitelisted.
This suggests that the QuantileDiscretizer class is restricted due to security policies within the current environment. Could you kindly advise on how to proceed
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-31-2024 06:41 AM
@sandeephenkel23 in which DBR release or environment are you facing this?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-04-2024 03:45 AM
Hi ,
I am using 13.3 LTS (includes Apache Spark 3.4.1, Scala 2.12)
Can you please help to resolve this?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-04-2024 04:05 AM
@sandeephenkel23 I've run the same code in DBR 13.3 LTS, and 1) It is successfully imported, 2) I can confirm it is in the whitelisted libs. Hence I'm wondering if there's anything else particular in your use case, triggering this? Is your use case a simple DBR 13.3 LTS bare cluster, no additional configs, and also no additional code other than what you have shared here, not used within a UDF, or in a py environment, etc, etc? If you can provide the exact reproducer, we can get to the root cause. Please consider all possible variables, like cluster mode (Assigned, Shared, etc), policies, spark properties, etc.
