cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

IllegalArgumentException: requirement failed: Result for RPC Some(e100cace-3836-4461-8902-80b3744fcb6b) lost, please retry your request.

JohanRex
New Contributor II

I'm using databricks connect to talk to a cluster on Azure. When doing a count on a dataframe I sometimes get this error message. Once I've gotten it once I don't seem to be able to get rid of it even if I restart my dev environment.

---------------------------------------------------------------------------
IllegalArgumentException                  Traceback (most recent call last)
c:\git\cap-mlv\notebooks\tmp_lookup_table.ipynb Cell 13' in <module>
----> 1 print(lookup_table_new.count())
 
File ~\AppData\Local\Continuum\anaconda3\envs\dbconnect\lib\site-packages\pyspark\sql\dataframe.py:670, in DataFrame.count(self)
    660 def count(self):
    661     """Returns the number of rows in this :class:`DataFrame`.
    662 
    663     .. versionadded:: 1.3.0
   (...)
    668     2
    669     """
--> 670     return int(self._jdf.count())
 
File ~\AppData\Local\Continuum\anaconda3\envs\dbconnect\lib\site-packages\py4j\java_gateway.py:1304, in JavaMember.__call__(self, *args)
   1298 command = proto.CALL_COMMAND_NAME +\
   1299     self.command_header +\
   1300     args_command +\
   1301     proto.END_COMMAND_PART
   1303 answer = self.gateway_client.send_command(command)
-> 1304 return_value = get_return_value(
   1305     answer, self.gateway_client, self.target_id, self.name)
   1307 for temp_arg in temp_args:
   1308     temp_arg._detach()
 
File ~\AppData\Local\Continuum\anaconda3\envs\dbconnect\lib\site-packages\pyspark\sql\utils.py:123, in capture_sql_exception.<locals>.deco(*a, **kw)
    119 converted = convert_exception(e.java_exception)
    120 if not isinstance(converted, UnknownException):
    121     # Hide where the exception came from that shows a non-Pythonic
    122     # JVM exception message.
--> 123     raise converted from None
    124 else:
    125     raise
 
IllegalArgumentException: requirement failed: Result for RPC Some(c4a9d516-8a33-4574-bf03-b3403c5d1b45) lost, please retry your request.

Using databricks runtime 9.1 LTS.

How can I narrow the problem down further?

1 ACCEPTED SOLUTION

Accepted Solutions

Anonymous
Not applicable

Hi @Johan Rexโ€‹ We checked with databricks connect team, this issue can happen when the library is too large to upload,

Databricks recommends that you use dbx by Databricks Labs for local development instead of Databricks Connect. Databricks plans no new feature development for Databricks Connect at this time. Also, be aware of the limitations of Databricks Connect.

https://docs.databricks.com/dev-tools/databricks-connect.html

https://docs.databricks.com/dev-tools/dbx.html

View solution in original post

3 REPLIES 3

Hubert-Dudek
Esteemed Contributor III

I think it is issue with databricks-connect. It run to some connections problem. Soon databricks tunnel will be available so you will run code from your IDE on databricks (currently it is on spark cluster)

JohanRex
New Contributor II

@Hubert Dudekโ€‹ Yes it's likely related to the communication with the cluster. But it would be great if it was possible to narrow it down more so I can possibly find a way forward. It's pretty much stopping me right now.

I'm looking forward to the tunnel but until then I need to get stuff done. ๐Ÿ˜•

Anonymous
Not applicable

Hi @Johan Rexโ€‹ We checked with databricks connect team, this issue can happen when the library is too large to upload,

Databricks recommends that you use dbx by Databricks Labs for local development instead of Databricks Connect. Databricks plans no new feature development for Databricks Connect at this time. Also, be aware of the limitations of Databricks Connect.

https://docs.databricks.com/dev-tools/databricks-connect.html

https://docs.databricks.com/dev-tools/dbx.html

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group