topic Re: Is it possible to pass configuration to a job on high concurrency cluster? in Data Engineering

Is it possible to pass configuration to a job on high concurrency cluster?

DouglasLinder — Wed, 22 Sep 2021 05:32:19 GMT

On a regular cluster, you can use:

```

spark.sparkContext._jsc.hadoopConfiguration().set(key, value)

```

These values are then available on the executors using the hadoop configuration.

However, on a high concurrency cluster, attempting to do so results in:

> py4j.security.Py4JSecurityException: Method public org.apache.hadoop.conf.Configuration org.apache.spark.api.java.JavaSparkContext.hadoopConfiguration() is not whitelisted on class class org.apache.spark.api.java.JavaSparkContext

Is there a way around this? Or is a limitation of the high concurrency cluster type?

The goal here is to pass tokens that are generate at runtime to the executor, which means that setting the details in the cluster settings (ie. cluster > advanced > spark > spark config) is not suitable.

Re: Is it possible to pass configuration to a job on high concurrency cluster?

Ryan_Chynoweth — Wed, 22 Sep 2021 20:21:31 GMT

I am not sure why you are getting that error on a high concurrency cluster. As I am able to set the configuration as you show above.

Can you try the following code instead?

sc._jsc.hadoopConfiguration().set(key, value)

Re: Is it possible to pass configuration to a job on high concurrency cluster?

DouglasLinder — Thu, 23 Sep 2021 00:36:37 GMT

Sorry, I failed to mention that I had selected 'credential passthrough' for my cluster, which adds the these features to the cluster configuration:

spark.databricks.passthrough.enabled true

spark.databricks.pyspark.enableProcessIsolation true

If you remove this then it works as normal.

Re: Is it possible to pass configuration to a job on high concurrency cluster?

Ryan_Chynoweth — Thu, 23 Sep 2021 15:38:15 GMT

Got it. I am fairly certain that is related to the passthrough limitations as you probably assumed. In the link it says there are limitations around SparkContext methods and SparkSession objects. It does not explicitly list your issue but I assume it is similar enough to apply.

Does removing those configs work for your environment? Do you require passthrough?

Re: Is it possible to pass configuration to a job on high concurrency cluster?

DouglasLinder — Tue, 28 Sep 2021 03:00:24 GMT

I guess it's just not supported; I wanted per-user hadoop configuration.