We run several workflows and tasks parallel using serverless compute. In many different places of code we started to get errors as below. It looks like that when one task fails, every other that run at the same moment fails as well. After retry on one by one, error disappears. Any ideas what can be the reason ?
/databricks/python/lib/python3.11/site-packages/pyspark/sql/connect/streaming/query.py:384: UserWarning: Failed to add the listener because of exception: <_MultiThreadedRendezvous of RPC that terminated with:
status = StatusCode.INTERNAL
details = "Cannot operate on a handle that is closed."
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2025-03-26T15:04:39.948318568+00:00", grpc_status:13, grpc_message:"Cannot operate on a handle that is closed."}"
>
The listener is not added, please add it again.
warnings.warn(f"ReleaseExecute failed with exception: {e}.")
OR
/databricks/python/lib/python3.11/site-packages/pyspark/sql/connect/client/reattach.py:253: UserWarning: ReleaseExecute failed with exception: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "[INVALID_HANDLE.SESSION_NOT_FOUND] The handle 5d3d1595-c845-4c0c-abf5-f731fa1bf93c is invalid. Session not found. SQLSTATE: HY000"
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2025-03-26T12:06:08.526677766+00:00", grpc_status:13, grpc_message:"[INVALID_HANDLE.SESSION_NOT_FOUND] The handle 5d3d1595-c845-4c0c-abf5-f731fa1bf93c is invalid. Session not found. SQLSTATE: HY000"}"
>.
warnings.warn(f"ReleaseExecute failed with exception: {e}.")
