topic Re: Connection reset by peer logging when importing custom package in Data Engineering

Connection reset by peer logging when importing custom package

analytics_eng — Thu, 09 Jan 2025 08:24:05 GMT

Hi! I'm trying to import a custom package I published to Azure Artifacts, but I keep seeing the INFO logging below, which I don't want to display. The package was installed correctly on the cluster, and it imports successfully, but the log still appears.Interestingly, the log does not appear when running the import (in notebook for example)inside a job. In the custom package, I use a logging function (based on the logging package) where I tried to suppress the 'Connection by peer' error, but it didn't work. With statements like:

logging.getLogger("py4j").setLevel(logging.ERROR)

logging.getLogger("py4j.java_gateway").setLevel(logging.ERROR)

logging.getLogger("py4j.clientserver").setLevel(logging.ERROR)

I also saw this post mentioning that the 'Connection reset by peer' log typically indicates that the connection between the client and the server was forcibly closed by the server. I tried increasing the memory, but this didn't work. Since importing the package doesn't do anything yet, apart from initializing the logger, I am stuck on how to stop showing this log

Any ideas or suggestions?

2025-01-09 08:12:26,679 - py4j.clientserver - INFO - Error while sending or receiving. Traceback (most recent call last): File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 528, in send_command self.socket.sendall(command.encode("utf-8")) ConnectionResetError: [Errno 104] Connection reset by peer 2025-01-09 08:12:26,684 - py4j.clientserver - INFO - Closing down clientserver connection 2025-01-09 08:12:26,694 - root - INFO - Exception while sending command. Traceback (most recent call last): File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 528, in send_command self.socket.sendall(command.encode("utf-8")) ConnectionResetError: [Errno 104] Connection reset by peer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1038, in send_command response = connection.send_command(command) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 531, in send_command raise Py4JNetworkError( py4j.protocol.Py4JNetworkError: Error while sending 2025-01-09 08:12:26,719 - py4j.clientserver - INFO - Closing down clientserver connection 2025-01-09 08:12:26,840 - py4j.clientserver - INFO - Error while sending or receiving. Traceback (most recent call last): File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 528, in send_command self.socket.sendall(command.encode("utf-8")) ConnectionResetError: [Errno 104] Connection reset by peer 2025-01-09 08:12:26,841 - py4j.clientserver - INFO - Closing down clientserver connection 2025-01-09 08:12:26,842 - root - INFO - Exception while sending command. Traceback (most recent call last): File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 528, in send_command self.socket.sendall(command.encode("utf-8")) ConnectionResetError: [Errno 104] Connection reset by peer During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1038, in send_command response = connection.send_command(command) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 531, in send_command raise Py4JNetworkError( py4j.protocol.Py4JNetworkError: Error while sending 2025-01-09 08:12:26,846 - py4j.clientserver - INFO - Closing down clientserver connection

Re: Connection reset by peer logging when importing custom package

Walter_C — Thu, 09 Jan 2025 09:56:46 GMT

Based on the context provided, here are some potential causes and solutions for the "Connection reset by peer" error

Network Issues: The error might be due to transient network issues. It is recommended to check the network stability and ensure that there are no interruptions or high latency.
Server Overload: The server might be overloaded and unable to handle the incoming connections. Monitoring the server's resource usage (CPU, memory, etc.) can help identify if this is the case.
Firewall/Proxy: A firewall or proxy might be terminating the connection. Ensure that the necessary ports are open and that there are no firewall rules blocking the connection.
Timeout Settings: Increasing the socket timeout settings might help if the connection is being reset due to timeouts. For example, setting spark.python.authenticate.socketTimeout to a higher value (e.g., 120s) might help.

Re: Connection reset by peer logging when importing custom package

analytics_eng — Mon, 20 Jan 2025 08:30:46 GMT

Thanks for the suggestions. I investigated all of the above, but they didn't provide a solution. What did work was using another logging package within my custom package: Loguru. Not sure why this helped?

Re: Connection reset by peer logging when importing custom package

AmeyJoshi — Tue, 18 Feb 2025 16:40:28 GMT

Please check if something like this works for you:
```

class IgnoreError(logging.Filter😞

def __init__(self, error_to_ignore😞

super().__init__()

self.error_to_ignore = error_to_ignore

def filter(self, record😞

return self.error_to_ignore not in record.getMessage()

```

Attach a handler to your `logger` to ignore this error:
```

handler = logging.StreamHandler()

handler.addFilter(IgnoreError("Exception while sending command"))

logger.handlers = [handler]

```

Re: Connection reset by peer logging when importing custom package

siklosib — Thu, 13 Mar 2025 23:10:47 GMT

What solved this problem for me is to remove the root logger configuration from the logging config and create another one within the loggers section. See below.

{
    'version': 1,
    'disable_existing_loggers': False,

    'formatters': {
        'simple': {
            'format': '%(message)s'
        },
    }

    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'level': 'INFO',
            'formatter': 'simple',
            'stream': 'ext://sys.stdout'
        },
    },

    'loggers': {
        #### use this instead in your app ####
        'my': {
            'level': 'INFO',
            'handlers': ['console'],
            'propagate': False,
        }
    },

    #### remove this ####
    'root': {
        'level': 'DEBUG',
        'handlers': ['console']
    },
}