Is it possible to pass configuration to a job on high concurrency cluster?

[DouglasLinder]

On a regular cluster, you can use:

```

spark.sparkContext._jsc.hadoopConfiguration().set(key, value)

```

These values are then available on the executors using the hadoop configuration.

However, on a high concurrency cluster, attempting to do so results in:

> py4j.security.Py4JSecurityException: Method public org.apache.hadoop.conf.Configuration org.apache.spark.api.java.JavaSparkContext.hadoopConfiguration() is not whitelisted on class class org.apache.spark.api.java.JavaSparkContext

Is there a way around this? Or is a limitation of the high concurrency cluster type?

The goal here is to pass tokens that are generate at runtime to the executor, which means that setting the details in the cluster settings (ie. cluster > advanced > spark > spark config) is not suitable.

[Kaniz]

Hi @ DouglasLinder! My name is Kaniz, and I'm the technical moderator here. Great to meet you, and thanks for your question! Let's see if your peers on the community have an answer to your question first. Or else I will follow up with my team and get back to you soon.Thanks.

[Ryan_Chynoweth]

I am not sure why you are getting that error on a high concurrency cluster. As I am able to set the configuration as you show above.

Can you try the following code instead?

sc._jsc.hadoopConfiguration().set(key, value)

[DouglasLinder]

Sorry, I failed to mention that I had selected 'credential passthrough' for my cluster, which adds the these features to the cluster configuration:

spark.databricks.passthrough.enabled true

spark.databricks.pyspark.enableProcessIsolation true

If you remove this then it works as normal.

[Ryan_Chynoweth]

Got it. I am fairly certain that is related to the passthrough limitations as you probably assumed. In the link it says there are limitations around SparkContext methods and SparkSession objects. It does not explicitly list your issue but I assume it is similar enough to apply.

Does removing those configs work for your environment? Do you require passthrough?

[DouglasLinder]

I guess it's just not supported; I wanted per-user hadoop configuration.