Driver is up but is not responsive, likely due to GC.

[vamsivarun007]

Hi all,

"Driver is up but is not responsive, likely due to GC."

This is the message in cluster event logs. Can anyone help me with this. What does GC means? Garbage collection? Can we control it externally?

[shyam_9]

Hi @vamsivarun007,

Please go through the below KB article to resolve this issue,https://kb.databricks.com/jobs/driver-unavailable.htmlfor what is GC, please check this answer,https://forums.databricks.com/questions/14725/how-to-resolve-spark-full-gc-on-cluster-startup.html

[Carlos_AlbertoG]

spark.catalog.clearCache() solve the problem for me 😉