ERROR - Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.

[pjp94]

I get the below error when trying to run multi-threading - fails towards the end of the run. My guess is it's related to memory/worker config. I've seen some solutions involving modifying the number of workers or CPU on the cluster - however that's not an option for me. If I can't change any cluster configs, what's the best way to resolve this error - decrease the max workers on my thread pool executor?

from concurrent.futures import ThreadPoolExecutor
 
with ThreadPoolExecutor() as executor:
  executor.map(func, arg)
  .
  .
  .
 
Error:
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1573.0 failed 4 times, most recent failure: Lost task 0.3 in stage 1573.0 (TID #####) (##.###.###.### executor 87): ExecutorLostFailure (executor 87 exited caused by one of the running tasks) Reason: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages.

[pjp94]

Since I don't have permissions to change cluster configurations, the only solution that ended up working was setting a max thread count to about half of the actual max so I don't overload the containers. However, open to any other optimization ideas!