I'm using databricks connect to talk to a cluster on Azure. When doing a count on a dataframe I sometimes get this error message. Once I've gotten it once I don't seem to be able to get rid of it even if I restart my dev environment.
---------------------------------------------------------------------------
IllegalArgumentException Traceback (most recent call last)
c:\git\cap-mlv\notebooks\tmp_lookup_table.ipynb Cell 13' in <module>
----> 1 print(lookup_table_new.count())
File ~\AppData\Local\Continuum\anaconda3\envs\dbconnect\lib\site-packages\pyspark\sql\dataframe.py:670, in DataFrame.count(self)
660 def count(self):
661 """Returns the number of rows in this :class:`DataFrame`.
662
663 .. versionadded:: 1.3.0
(...)
668 2
669 """
--> 670 return int(self._jdf.count())
File ~\AppData\Local\Continuum\anaconda3\envs\dbconnect\lib\site-packages\py4j\java_gateway.py:1304, in JavaMember.__call__(self, *args)
1298 command = proto.CALL_COMMAND_NAME +\
1299 self.command_header +\
1300 args_command +\
1301 proto.END_COMMAND_PART
1303 answer = self.gateway_client.send_command(command)
-> 1304 return_value = get_return_value(
1305 answer, self.gateway_client, self.target_id, self.name)
1307 for temp_arg in temp_args:
1308 temp_arg._detach()
File ~\AppData\Local\Continuum\anaconda3\envs\dbconnect\lib\site-packages\pyspark\sql\utils.py:123, in capture_sql_exception.<locals>.deco(*a, **kw)
119 converted = convert_exception(e.java_exception)
120 if not isinstance(converted, UnknownException):
121 # Hide where the exception came from that shows a non-Pythonic
122 # JVM exception message.
--> 123 raise converted from None
124 else:
125 raise
IllegalArgumentException: requirement failed: Result for RPC Some(c4a9d516-8a33-4574-bf03-b3403c5d1b45) lost, please retry your request.
Using databricks runtime 9.1 LTS.
How can I narrow the problem down further?
