Hi all!
Recently we've been getting lots of these errors when running Databricks notebooks:
At that time we observed DRIVER_NOT_RESPONDING (Driver is up but is not responsive, likely due to GC.) log on the single-user cluster we use.
Previously when this error appeared in cluster logs it was due to 2 things:
- The number of notebooks attached to that cluster was above the 145 limit, or
- The cluster driver memory was exhausted.
Lately, it seems that neither of these two things happens but our notebooks still fail.
Do you have any idea what might be the problem here?
