Connection reset by peer logging when importing custom package

[analytics_eng]

Hi! I'm trying to import a custom package I published to Azure Artifacts, but I keep seeing the INFO logging below, which I don't want to display. The package was installed correctly on the cluster, and it imports successfully, but the log still appears.Interestingly, the log does not appear when running the import (in notebook for example)inside a job. In the custom package, I use a logging function (based on the logging package) where I tried to suppress the 'Connection by peer' error, but it didn't work. With statements like:

logging.getLogger("py4j").setLevel(logging.ERROR)

logging.getLogger("py4j.java_gateway").setLevel(logging.ERROR)

logging.getLogger("py4j.clientserver").setLevel(logging.ERROR)

I also saw this postPy4JJavaError: An error occurred while calling o552.count()  mentioning that the 'Connection reset by peer' log typically indicates that the connection between the client and the server was forcibly closed by the server. I tried increasing the memory, but this didn't work. Since importing the package doesn't do anything yet, apart from initializing the logger, I am stuck on how to stop showing this log

Any ideas or suggestions?

2025-01-09 08:12:26,679 - py4j.clientserver - INFO - Error while sending or receiving.
Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 528, in send_command
    self.socket.sendall(command.encode("utf-8"))
ConnectionResetError: [Errno 104] Connection reset by peer
2025-01-09 08:12:26,684 - py4j.clientserver - INFO - Closing down clientserver connection
2025-01-09 08:12:26,694 - root - INFO - Exception while sending command.
Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 528, in send_command
    self.socket.sendall(command.encode("utf-8"))
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1038, in send_command
    response = connection.send_command(command)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 531, in send_command
    raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending
2025-01-09 08:12:26,719 - py4j.clientserver - INFO - Closing down clientserver connection
2025-01-09 08:12:26,840 - py4j.clientserver - INFO - Error while sending or receiving.
Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 528, in send_command
    self.socket.sendall(command.encode("utf-8"))
ConnectionResetError: [Errno 104] Connection reset by peer
2025-01-09 08:12:26,841 - py4j.clientserver - INFO - Closing down clientserver connection
2025-01-09 08:12:26,842 - root - INFO - Exception while sending command.
Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 528, in send_command
    self.socket.sendall(command.encode("utf-8"))
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1038, in send_command
    response = connection.send_command(command)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 531, in send_command
    raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending
2025-01-09 08:12:26,846 - py4j.clientserver - INFO - Closing down clientserver connection

[Walter_C]

Based on the context provided, here are some potential causes and solutions for the "Connection reset by peer" error

1. Network Issues: The error might be due to transient network issues. It is recommended to check the network stability and ensure that there are no interruptions or high latency.

2. Server Overload: The server might be overloaded and unable to handle the incoming connections. Monitoring the server's resource usage (CPU, memory, etc.) can help identify if this is the case.

3. Firewall/Proxy: A firewall or proxy might be terminating the connection. Ensure that the necessary ports are open and that there are no firewall rules blocking the connection.

4. Timeout Settings: Increasing the socket timeout settings might help if the connection is being reset due to timeouts. For example, setting spark.python.authenticate.socketTimeout to a higher value (e.g., 120s) might help.

[analytics_eng]

Thanks for the suggestions. I investigated all of the above, but they didn't provide a solution. What did work was using another logging package within my custom package: Loguru. Not sure why this helped?

[AmeyJoshi]

Please check if something like this works for you:
```

class IgnoreError(logging.Filter😞

    def __init__(self, error_to_ignore😞

        super().__init__()

        self.error_to_ignore = error_to_ignore

    def filter(self, record😞

        return self.error_to_ignore not in record.getMessage()

```

Attach a handler to your `logger` to ignore this error:
```

handler = logging.StreamHandler()

handler.addFilter(IgnoreError("Exception while sending command"))

logger.handlers = [handler]

```

[siklosib]

What solved this problem for me is to remove the root logger configuration from the logging config and create another one within the loggers section. See below.

{
    'version': 1,
    'disable_existing_loggers': False,

    'formatters': {
        'simple': {
            'format': '%(message)s'
        },
    }

    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'level': 'INFO',
            'formatter': 'simple',
            'stream': 'ext://sys.stdout'
        },
    },

    'loggers': {
        #### use this instead in your app ####
        'my': {
            'level': 'INFO',
            'handlers': ['console'],
            'propagate': False,
        }
    },

    #### remove this ####
    'root': {
        'level': 'DEBUG',
        'handlers': ['console']
    },
}