@ls thanks for your question!

Since this is a PySpark application, the "Connection reset by peer" error seems to mask the actual exception. This type of issue is often linked to memory problems where Python workers are terminated, so the JVM <-> Python connection is reset. Here are some suggestions:

• Memory Profiling: Try freezing the dataset that reproduces the problem and profile memory usage on the Python side. Look for issues like data distribution problems or skewness. Tools like memory-profiler might help.

• Error Isolation: If possible, isolate the query and dataset causing the issue. Translating the Python code to Scala can help revealing the underlying exception.

• Logs and Metrics: Check if there are any OOM-related messages in the executor logs. Additionally, analyze task metrics per executor in the Spark UI for any anomalies.

• Cluster Resources: As a last resort, you could temporarily increase cluster resources to allow the job to complete. This approach can help you gather insights from the Spark UI metrics to understand why the current cluster size is insufficient for processing the dataset.

Additional Resources:

https://pypi.org/project/memory-profiler/ 

https://www.databricks.com/blog/2022/11/30/memory-profiling-pyspark.html