Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-26-2024 07:56 AM - edited 03-26-2024 07:59 AM
Thank you for your reply, @Retired_mod . But there is no issue in the availability of databricks-connect. I had a bit time to look into it and found that this issue does not exist in databricks-connect with a custer with runtime 13.3. It occurs with databricks-connect 14.3 and a cluster with Runtime 14.3.
databricks-connect-13.3 and Runtime 13.3 Cluster:
from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.profile("DEBUGGING_133").getOrCreate()
spark.sql("SELECT 1")
# output: DataFrame[1: int]
# >>> Databricks cluster shuts down (e.g. because of timeout because of long running script)
spark.sql("SELECT 1")
# Cluster starts again automatically
# output: DataFrame[1: int]
databricks-connect-14.3 and Runtime 14.3 Cluster:
from databricks.connect import DatabricksSession
spark = DatabricksSession.builder.profile("DEBUGGING_133").getOrCreate()
spark.sql("SELECT 1")
# output: DataFrame[1: int]
# >>> Databricks cluster shuts down (e.g. because of timeout because of long running script)
spark.sql("SELECT 1")
# Cluster starts again automatically
# output:
Traceback (most recent call last):
File "****\lib\site-packages\IPython\core\interactiveshell.py", line 3508, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-7-e8eb9b165388>", line 1, in <module>
spark.sql("SELECT 1")
File "****\lib\site-packages\pyspark\sql\connect\session.py", line 572, in sql
data, properties = self.client.execute_command(cmd.command(self._client))
File "****\lib\site-packages\pyspark\sql\connect\client\core.py", line 1139, in execute_command
data, _, _, _, properties = self._execute_and_fetch(req, observations or {})
File "****\lib\site-packages\pyspark\sql\connect\client\core.py", line 1515, in _execute_and_fetch
for response in self._execute_and_fetch_as_iterator(req, observations):
File "****\lib\site-packages\pyspark\sql\connect\client\core.py", line 1493, in _execute_and_fetch_as_iterator
self._handle_error(error)
File "****\lib\site-packages\pyspark\sql\connect\client\core.py", line 1805, in _handle_error
raise error
File "****\lib\site-packages\pyspark\sql\connect\client\core.py", line 1486, in _execute_and_fetch_as_iterator
yield from handle_response(b)
File "****\lib\site-packages\pyspark\sql\connect\client\core.py", line 1406, in handle_response
self._verify_response_integrity(b)
File "****\lib\site-packages\pyspark\sql\connect\client\core.py", line 1937, in _verify_response_integrity
raise PySparkAssertionError(
pyspark.errors.exceptions.base.PySparkAssertionError: Received incorrect server side session identifier for request. Please create a new Spark Session to reconnect. (ab413162-708a-423f-84c7-b04969ed3bf4 != 3c8ea3e4-e20f-4a31-82a0-ff938f4017c6)
Is this maybe a bug? Where can I see known issues or report this?