Connection reset by peer logging when importing custom package

[analytics_eng]

Hi! I'm trying to import a custom package I published to Azure Artifacts, but I keep seeing the INFO logging below, which I don't want to display. The package was installed correctly on the cluster, and it imports successfully, but the log still appears.Interestingly, the log does not appear when running the import (in notebook for example)inside a job. In the custom package, I use a logging function (based on the logging package) where I tried to suppress the 'Connection by peer' error, but it didn't work. With statements like:

logging.getLogger("py4j").setLevel(logging.ERROR)

logging.getLogger("py4j.java_gateway").setLevel(logging.ERROR)

logging.getLogger("py4j.clientserver").setLevel(logging.ERROR)

I also saw this postPy4JJavaError: An error occurred while calling o552.count()  mentioning that the 'Connection reset by peer' log typically indicates that the connection between the client and the server was forcibly closed by the server. I tried increasing the memory, but this didn't work. Since importing the package doesn't do anything yet, apart from initializing the logger, I am stuck on how to stop showing this log

Any ideas or suggestions?

2025-01-09 08:12:26,679 - py4j.clientserver - INFO - Error while sending or receiving.
Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 528, in send_command
    self.socket.sendall(command.encode("utf-8"))
ConnectionResetError: [Errno 104] Connection reset by peer
2025-01-09 08:12:26,684 - py4j.clientserver - INFO - Closing down clientserver connection
2025-01-09 08:12:26,694 - root - INFO - Exception while sending command.
Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 528, in send_command
    self.socket.sendall(command.encode("utf-8"))
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1038, in send_command
    response = connection.send_command(command)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 531, in send_command
    raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending
2025-01-09 08:12:26,719 - py4j.clientserver - INFO - Closing down clientserver connection
2025-01-09 08:12:26,840 - py4j.clientserver - INFO - Error while sending or receiving.
Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 528, in send_command
    self.socket.sendall(command.encode("utf-8"))
ConnectionResetError: [Errno 104] Connection reset by peer
2025-01-09 08:12:26,841 - py4j.clientserver - INFO - Closing down clientserver connection
2025-01-09 08:12:26,842 - root - INFO - Exception while sending command.
Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 528, in send_command
    self.socket.sendall(command.encode("utf-8"))
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1038, in send_command
    response = connection.send_command(command)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/databricks/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/clientserver.py", line 531, in send_command
    raise Py4JNetworkError(
py4j.protocol.Py4JNetworkError: Error while sending
2025-01-09 08:12:26,846 - py4j.clientserver - INFO - Closing down clientserver connection

[Walter_C]

Based on the context provided, here are some potential causes and solutions for the "Connection reset by peer" error

1. Network Issues: The error might be due to transient network issues. It is recommended to check the network stability and ensure that there are no interruptions or high latency.

2. Server Overload: The server might be overloaded and unable to handle the incoming connections. Monitoring the server's resource usage (CPU, memory, etc.) can help identify if this is the case.

3. Firewall/Proxy: A firewall or proxy might be terminating the connection. Ensure that the necessary ports are open and that there are no firewall rules blocking the connection.

4. Timeout Settings: Increasing the socket timeout settings might help if the connection is being reset due to timeouts. For example, setting spark.python.authenticate.socketTimeout to a higher value (e.g., 120s) might help.