topic OSError: [Errno 78] Remote address changed in Get Started Discussions

OSError: [Errno 78] Remote address changed

liormayn — Thu, 18 Apr 2024 11:09:11 GMT

Hello:)

as part of deploying an app that previously ran directly on emr to databricks, we are running experiments using LTS 9.1, and getting the following error:

PythonException: An exception was thrown from a UDF: 'pyspark.serializers.SerializationError: Caused by Traceback (most recent call last): File "/databricks/spark/python/pyspark/serializers.py", line 165, in _read_with_length return self.loads(obj) File "/databricks/spark/python/pyspark/serializers.py", line 469, in loads return pickle.loads(obj, encoding=encoding) File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 844, in exec_module File "<frozen importlib._bootstrap_external>", line 980, in get_code File "<frozen importlib._bootstrap_external>", line 1038, in get_data OSError: [Errno 78] Remote address changed'. Full traceback below: Traceback (most recent call last): File "/databricks/spark/python/pyspark/serializers.py", line 165, in _read_with_length return self.loads(obj) File "/databricks/spark/python/pyspark/serializers.py", line 469, in loads return pickle.loads(obj, encoding=encoding) File "<frozen importlib._bootstrap>", line 991, in _find_and_load File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 671, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 844, in exec_module File "<frozen importlib._bootstrap_external>", line 980, in get_code File "<frozen importlib._bootstrap_external>", line 1038, in get_data OSError: [Errno 78] Remote address changed

the same udf works in emr, and is part of the codebase that databricks uses as git source,

did anyone encounter this error?

Would appreciate any advice in troubleshooting it, Thanks!

Re: OSError: [Errno 78] Remote address changed

shan_chandra — Thu, 18 Apr 2024 22:46:13 GMT

@liormayn - could you please let us know if you had a chance to run it on DBR 10.4 LTS?

Re: OSError: [Errno 78] Remote address changed

liormayn — Sun, 21 Apr 2024 07:24:21 GMT

yes:) @shan_chandra this is what happened when i tried 10.4 lts :
https://community.databricks.com/t5/community-discussions/error-while-encoding-java-lang-runtimeexception-org-apache-spark/td-p/66586

Re: OSError: [Errno 78] Remote address changed

NandiniN — Wed, 01 May 2024 08:40:55 GMT

Hi @liormayn ,

Are you still facing the issue? This was faced in mid March and issue was fixed. It can happen for some pip install when the libraries are in Workspace. But if you are still facing the issue, I would suggest you to create a support ticket for this as it may need involved investigation.

Re: OSError: [Errno 78] Remote address changed

liormayn — Thu, 02 May 2024 13:33:02 GMT

Hey @NandiniN 🙂 The error currently stopped happening, but we are not feeling "safe" yet,
could you tell me when the fix was published? just so we try and pin point to see if the fix is what solved it?

Re: OSError: [Errno 78] Remote address changed

NandiniN — Sun, 05 May 2024 17:15:13 GMT

Hi @liormayn ,

I can understand. I see the fix went on 20 March 2024, you would have to restart the clusters.

Thanks!