topic Re: Databricks Cluster job failure issue in Get Started Discussions

Databricks Cluster job failure issue

PKD28 — Thu, 05 Sep 2024 11:52:34 GMT

Jobs within the all purpose DB Cluster are failing with "the spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached"

In the event log it says "Event_type=DRIVER_NOT_RESPONDING & MESSAGE= "Driver is up but is not responsive, likely due to GC."

Please help me to fix this.

Re: Databricks Cluster job failure issue

szymon_dybczak — Thu, 05 Sep 2024 13:03:00 GMT

Hi @PKD28 ,

One common cause for this error is that the driver is undergoing a memory bottleneck. When this happens, the driver crashes with an out of memory (OOM) condition and gets restarted or becomes unresponsive due to frequent full garbage collection. So, 9/10 times GC is due to out of memory exceptions. What you can try to do is to increase drivers memory first and see if that helps.

Re: Databricks Cluster job failure issue

PKD28 — Thu, 05 Sep 2024 13:29:05 GMT

just now there is one cluster issue

cluster error: Driver is unresponsive likely due to GC

cluster conf:

worker: Standard_D8ads_v5

Driver: standard_E16d_v4

What do you suggest here ??

Re: Databricks Cluster job failure issue

szymon_dybczak — Thu, 05 Sep 2024 13:43:11 GMT

I would try to use driver with higher amount of memory, just to check if it will handle the load. So maybe I'll try to run a process on Standard_E20d_v4 or Standard_E32d_v4 (this one has 2x more RAM memory, so it should work)