cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

OSError: [Errno 78] Remote address changed

liormayn
New Contributor III

Hello:)

as part of deploying an app that previously ran directly on emr to databricks, we are running experiments using LTS 9.1, and getting the following error:

 

PythonException: An exception was thrown from a UDF: 'pyspark.serializers.SerializationError: Caused by Traceback (most recent call last):
  File "/databricks/spark/python/pyspark/serializers.py", line 165, in _read_with_length
    return self.loads(obj)
  File "/databricks/spark/python/pyspark/serializers.py", line 469, in loads
    return pickle.loads(obj, encoding=encoding)
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 844, in exec_module
  File "<frozen importlib._bootstrap_external>", line 980, in get_code
  File "<frozen importlib._bootstrap_external>", line 1038, in get_data
OSError: [Errno 78] Remote address changed'. Full traceback below:
Traceback (most recent call last):
  File "/databricks/spark/python/pyspark/serializers.py", line 165, in _read_with_length
    return self.loads(obj)
  File "/databricks/spark/python/pyspark/serializers.py", line 469, in loads
    return pickle.loads(obj, encoding=encoding)
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 844, in exec_module
  File "<frozen importlib._bootstrap_external>", line 980, in get_code
  File "<frozen importlib._bootstrap_external>", line 1038, in get_data
OSError: [Errno 78] Remote address changed

 

 the same udf works in emr, and is part of the codebase that databricks uses as git source,

did anyone encounter this error? 

Would appreciate any advice in troubleshooting it, Thanks!

5 REPLIES 5

shan_chandra
Esteemed Contributor

@liormayn  - could you please let us know if you had a chance to run it on DBR 10.4 LTS?

NandiniN
Honored Contributor

Hi @liormayn , 

Are you still facing the issue? This was faced in mid March and issue was fixed. It can happen for some pip install when the libraries are in Workspace. But if you are still facing the issue, I would suggest you to create a support ticket for this as it may need involved investigation.

liormayn
New Contributor III

Hey @NandiniN ๐Ÿ™‚ The error currently stopped happening, but we are not feeling "safe" yet,
could you tell me when the fix was published? just so we try and pin point to see if the fix is what solved it? 

NandiniN
Honored Contributor

Hi @liormayn ,

I can understand. I see the fix went on 20 March 2024, you would have to restart the clusters.

Thanks!

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group