Alberto_Umana
Databricks Employee
Databricks Employee

Yes, it is actually possible that driver due to some reason did not free up memory.. if that happens the you will see these kind of failures:

 

  • Unresponsiveness: The driver may become unresponsive, leading to failed health checks and potential restarts or kills by watchdog mechanisms.
  • Frequent Restarts: Continuous memory pressure and GC overhead can cause the driver to restart frequently, leading to interruptions in job executions and degraded performance.
  • Out of Memory (OOM) Conditions: Eventually, the driver might run out of memory, leading to crashes and job failures with explicit OOM errors