Hello,
I have a job cluster running streaming job and it unexpectedly failed on 19th March due to DRIVER_UNAVAILABLE (Request timed out, Driver is temporarily unavailable) in event log. This is the run: https://atlassian-discover.cloud.databricks.com/jobs/323849284041517/runs/395169892801478?o=44820012...
I'm aware of a thread reporting the same problem: https://kb.databricks.com/en_US/jobs/driver-unavailable and it pointed out memory pressure is a common cause. However, according to driver stdout there were only minor GCs that took around 30ms-40ms around the time the driver became unavailable:
![duliu_0-1712192893352.png duliu_0-1712192893352.png](/t5/image/serverpage/image-id/6902iF80733D7E348156E/image-size/medium/is-moderation-mode/true?v=v2&px=400)
I also checked the driver log (log4j logs) and it doesn't have any error messages, a few warning messages are unrelated. In fact the driver even continued outputting logs several minutes after the DRIVER_UNAVAILABLE error message appeared in event log.
I tried loading spark UI but after a long wait with messages saying processing files, it errors with the following message, so I can't see spark history UI as well:
![duliu_1-1712192913524.png duliu_1-1712192913524.png](/t5/image/serverpage/image-id/6903i7E36ECA8F5D2D1C9/image-size/medium/is-moderation-mode/true?v=v2&px=400)
Could anyone help please?