Driver is up but is not responsive, likely due to GC.

[Avi759787]

I am using Interactive cluster to run frequent (every 15min) batch job.

After certain time (example: 6hours), the cluster continuously starts showing Driver is up but is not responsive, likely due to GC. in event log and all jobs starts failing.

If the cluster is restarted all jobs again starts to execute successfully.

Can someone help here with what is the root cause and  how to resolve this without restarting cluster.

Thanks in Advance.

#drivernotresponsive #GC

[Kaniz]

Hi @Avi759787, 

• One straightforward solution is to increase the amount of memory on the driver node. You can adjust this configuration in your cluster settings. More memory allows the driver to handle larger workloads without running into memory constraints.
• You can also fine-tune GC settings to optimize memory management. However, this requires deeper knowledge of Spark internals.
• Sometimes, cached data can consume excessive memory. You can try clearing the cache using spark.catalog.clearCache()¹.