Run failed with error message ContextNotFound

[Karlo_Kotarac]

Hi all!

Recently we've been getting lots of these errors when running Databricks notebooks:

At that time we observed DRIVER_NOT_RESPONDING (Driver is up but is not responsive, likely due to GC.) log on the single-user cluster we use.
Previously when this error appeared in cluster logs it was due to 2 things:
- The number of notebooks attached to that cluster was above the 145 limit, or
- The cluster driver memory was exhausted.

Lately, it seems that neither of these two things happens but our notebooks still fail.

Do you have any idea what might be the problem here?

[jose_gonzalez]

are you able to get the full error stack trace from the driver's logs? 

[Karlo_Kotarac]

Unfortunately not for this event, but I will investigate these logs carefully next time when this error happens. Thnx!

[Lakshay]

You may also try to run the failing notebook on the job cluster

[Karlo_Kotarac]

In case somebody else runs into the same issue: After investigation from Databricks support the conclusion was that the driver's memory was overloaded ('Driver Not Responding' error message in the event log) but it can happen that we don't get the complete metric and Spark UI details when this happens. This is probably why I didn't see in the metrics tab that driver memory was exhausted. The suggested solution can be found in the following link https://kb.databricks.com/en_US/jobs/driver-unavailable