cancel
Showing results for 
Search instead for 
Did you mean: 
Community Discussions
cancel
Showing results for 
Search instead for 
Did you mean: 

OSError: [Errno 78] Remote address changed

liormayn
New Contributor III

Hello:)

as part of deploying an app that previously ran directly on emr to databricks, we are running experiments using LTS 9.1, and getting the following error:

 

PythonException: An exception was thrown from a UDF: 'pyspark.serializers.SerializationError: Caused by Traceback (most recent call last):
  File "/databricks/spark/python/pyspark/serializers.py", line 165, in _read_with_length
    return self.loads(obj)
  File "/databricks/spark/python/pyspark/serializers.py", line 469, in loads
    return pickle.loads(obj, encoding=encoding)
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 844, in exec_module
  File "<frozen importlib._bootstrap_external>", line 980, in get_code
  File "<frozen importlib._bootstrap_external>", line 1038, in get_data
OSError: [Errno 78] Remote address changed'. Full traceback below:
Traceback (most recent call last):
  File "/databricks/spark/python/pyspark/serializers.py", line 165, in _read_with_length
    return self.loads(obj)
  File "/databricks/spark/python/pyspark/serializers.py", line 469, in loads
    return pickle.loads(obj, encoding=encoding)
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 844, in exec_module
  File "<frozen importlib._bootstrap_external>", line 980, in get_code
  File "<frozen importlib._bootstrap_external>", line 1038, in get_data
OSError: [Errno 78] Remote address changed

 

 the same udf works in emr, and is part of the codebase that databricks uses as git source,

did anyone encounter this error? 

Would appreciate any advice in troubleshooting it, Thanks!

5 REPLIES 5

shan_chandra
Esteemed Contributor
Esteemed Contributor

@liormayn  - could you please let us know if you had a chance to run it on DBR 10.4 LTS?

NandiniN
Valued Contributor III
Valued Contributor III

Hi @liormayn , 

Are you still facing the issue? This was faced in mid March and issue was fixed. It can happen for some pip install when the libraries are in Workspace. But if you are still facing the issue, I would suggest you to create a support ticket for this as it may need involved investigation.

liormayn
New Contributor III

Hey @NandiniN 🙂 The error currently stopped happening, but we are not feeling "safe" yet,
could you tell me when the fix was published? just so we try and pin point to see if the fix is what solved it? 

NandiniN
Valued Contributor III
Valued Contributor III

Hi @liormayn ,

I can understand. I see the fix went on 20 March 2024, you would have to restart the clusters.

Thanks!