I am running a pipeline job in Databricks and it failed with the following message:
Run failed with error message Could not reach driver of cluster 5824-145411-p65jt7uo.
This message is not very descriptive, and I am not able to identify the root cause.
Could you please advise how I can access more detailed error logs or driver logs in such cases? Is there a way to capture the underlying error message that caused the cluster driver to become unreachable, so that I can investigate further?
Any guidance or best practices would be greatly appreciated.
Thank you!
Hi @minhhung0507 ,
Typically this error could appear when there's a high load on the driver node. Another reason could be related to high garbage collection on driver node as well as high memory and cpu which leads to throttling, and prevents the driver from responding within the allocated time.
Job run fails with error message “Could not reach driver of cluster” - Databricks
Workflows are failing with a 'Could not reach driver of the cluster' error - Databricks
As about accessing driver logs, in DLT you can access them in following way:
