topic Re: Driver is up but is not responsive, likely due to GC. in Data Engineering

Driver is up but is not responsive, likely due to GC.

vamsivarun007 — Fri, 13 Mar 2020 08:53:24 GMT

Hi all,

"Driver is up but is not responsive, likely due to GC."

This is the message in cluster event logs. Can anyone help me with this. What does GC means? Garbage collection? Can we control it externally?

Re: Driver is up but is not responsive, likely due to GC.

shyam_9 — Tue, 17 Mar 2020 19:59:32 GMT

Hi @vamsivarun007,

Please go through the below KB article to resolve this issue,

https://kb.databricks.com/jobs/driver-unavailable.html

for what is GC, please check this answer,

https://forums.databricks.com/questions/14725/how-to-resolve-spark-full-gc-on-cluster-startup.html

Re: Driver is up but is not responsive, likely due to GC.

Carlos_AlbertoG — Thu, 26 Nov 2020 07:57:30 GMT

spark.catalog.clearCache() solve the problem for me 😉

Re: Driver is up but is not responsive, likely due to GC.

Jaron — Tue, 18 Jun 2024 01:56:40 GMT

Hi, I meet the seme problem when I train a DeepLearning model. Could you tell me where to set this 'spark.catalog.clearCache()'? Thanks!

Re: Driver is up but is not responsive, likely due to GC.

jacovangelder — Tue, 18 Jun 2024 07:12:53 GMT

9/10 times GC is due to out of memory exceptions.

@Jaron spark.catalog.clearCache() is not a configurable option, but rather a command to submit.

Re: Driver is up but is not responsive, likely due to GC.

Jaron — Tue, 18 Jun 2024 07:22:55 GMT

Thanks! But I'm running the python script file via workflow→jobs, so I can't submit "spark.catalog.clearCache()" via notebooks because they're isolated. Is there any way out of this situation?😭.

For another question, may I ask if the ''memory'' you mentioned is spark.executor.memory? My program is running with 64GB of computer memory which is large enough, but still this GC issue occurs. I checked the docs and they all mention that it could be that the ''spark.executor.memory'' is too small, but I don't know how to check and deal with it. (so tired 😫

Looking forward your reply, thanks !!!