Is it possible to pass configuration to a job on high concurrency cluster?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-21-2021 10:32 PM
On a regular cluster, you can use:
```
spark.sparkContext._jsc.hadoopConfiguration().set(key, value)
```
These values are then available on the executors using the hadoop configuration.
However, on a high concurrency cluster, attempting to do so results in:
> py4j.security.Py4JSecurityException: Method public org.apache.hadoop.conf.Configuration org.apache.spark.api.java.JavaSparkContext.hadoopConfiguration() is not whitelisted on class class org.apache.spark.api.java.JavaSparkContext
Is there a way around this? Or is a limitation of the high concurrency cluster type?
The goal here is to pass tokens that are generate at runtime to the executor, which means that setting the details in the cluster settings (ie. cluster > advanced > spark > spark config) is not suitable.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-22-2021 01:21 PM
I am not sure why you are getting that error on a high concurrency cluster. As I am able to set the configuration as you show above.
Can you try the following code instead?
sc._jsc.hadoopConfiguration().set(key, value)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-22-2021 05:36 PM
Sorry, I failed to mention that I had selected 'credential passthrough' for my cluster, which adds the these features to the cluster configuration:
spark.databricks.passthrough.enabled true
spark.databricks.pyspark.enableProcessIsolation true
If you remove this then it works as normal.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-23-2021 08:38 AM
Got it. I am fairly certain that is related to the passthrough limitations as you probably assumed. In the link it says there are limitations around SparkContext methods and SparkSession objects. It does not explicitly list your issue but I assume it is similar enough to apply.
Does removing those configs work for your environment? Do you require passthrough?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-27-2021 08:00 PM
I guess it's just not supported; I wanted per-user hadoop configuration.
![](/skins/images/97567C72181EBE789E1F0FD869E4C89B/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/97567C72181EBE789E1F0FD869E4C89B/responsive_peak/images/icon_anonymous_message.png)