GC Allocation Failure

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

There are a couple of related posts here and here.
Seeing a similar issue with a long running job. Processes are in a "RUNNING" state, cluster is active, but stdout log shows the dreaded GC Allocation Failure.

Env:

I've set the following on the config:

.config("spark.cleaner.referenceTracking.cleanCheckpoints", "true")

.config("spark.cleaner.periodicGC.interval", "1min")

and have attempted to clear the cache:

spark.catalog.clearCache()

Is there anything else I can try? Is it possible to set up an alert for this error to kill the job when it enters this state so we aren't burning through resources?