topic Re: Run failed with error message ContextNotFound in Data Engineering

Run failed with error message ContextNotFound

Karlo_Kotarac — Thu, 18 Apr 2024 06:48:48 GMT

Hi all!

Recently we've been getting lots of these errors when running Databricks notebooks:

At that time we observed DRIVER_NOT_RESPONDING (Driver is up but is not responsive, likely due to GC.) log on the single-user cluster we use.
Previously when this error appeared in cluster logs it was due to 2 things:
- The number of notebooks attached to that cluster was above the 145 limit, or
- The cluster driver memory was exhausted.

Lately, it seems that neither of these two things happens but our notebooks still fail.

Do you have any idea what might be the problem here?

Re: Run failed with error message ContextNotFound

jose_gonzalez — Thu, 18 Apr 2024 21:44:46 GMT

are you able to get the full error stack trace from the driver's logs?

Re: Run failed with error message ContextNotFound

Karlo_Kotarac — Fri, 19 Apr 2024 08:30:10 GMT

Unfortunately not for this event, but I will investigate these logs carefully next time when this error happens. Thnx!

Re: Run failed with error message ContextNotFound

Lakshay — Fri, 19 Apr 2024 11:12:38 GMT

You may also try to run the failing notebook on the job cluster

Re: Run failed with error message ContextNotFound

Karlo_Kotarac — Fri, 07 Jun 2024 06:52:44 GMT

In case somebody else runs into the same issue: After investigation from Databricks support the conclusion was that the driver's memory was overloaded ('Driver Not Responding' error message in the event log) but it can happen that we don't get the complete metric and Spark UI details when this happens. This is probably why I didn't see in the metrics tab that driver memory was exhausted. The suggested solution can be found in the following link https://kb.databricks.com/en_US/jobs/driver-unavailable