09-22-2021 01:21 PM
I am not sure why you are getting that error on a high concurrency cluster. As I am able to set the configuration as you show above.
Can you try the following code instead?
sc._jsc.hadoopConfiguration().set(key, value)
