NandiniN
Databricks Employee
Databricks Employee

It may not necessarily be a bug, but some tuning due to architectural differences.

What the message says is:

  • The system was processing a gRPC operation identified by opId=5ef071b7-xxx, and it set a deadline for that operation (likely 120 seconds).
  • The operation didn't complete in time and exceeded the deadline, so the system has shut down the stream and stopped waiting for further results.
  • The operation spent almost all of its time (around 120 seconds) waiting for results and did not spend any time in the process of sending data back to the client.
  • It is an INFO message, indicating an event.
  • Shared mode uses spark connect underlying, but single user mode does not and hence we do not see the logs in single user cluster.

However, as our next steps:

 

  • We can try to understand the cause of the delay on the external resource or service where the request is sent. 
  • It is possible the timeouts are not same, or needs to be bumped up.
  • Are there any other errors that you see along with these messages?

 

However, yes it may need more indepth look.