How do you debug OOM on long-running streaming jobs?
Hi All,
One of my Structured Streaming jobs has started dying with OutOfMemoryError. Runs fine for about approx 40 hours, then falls over, restarts, and does the same thing again. I had introduced few changes between stable and failed runs but changes are not obvious(Of course, i started slicing changes and testing them in isolation).
I've poked at it for a couple of days now. Spark UI first, obviously. Checked the Storage tab in case something cached was growing, looked at heap usage in the compute metrics to see whether it's a slow climb or a sudden spike (it's a climb), and read the executor logs around the crash. GC was getting progressively worse before it went.
My main suspect right now is leaking at Native Memory not Heap or Stateful Store etc
Where I'm stuck is knowing when to stop looking and just give it more memory. I could bump memoryOverhead tomorrow and the job would probably survive another week!!! How do you tell the difference? Is there a signal you look for that says `this is genuinely a leak` vs `this job was always borderline`?
Also curious whether anyone bothers with heap dumps on Databricks. Every time I've considered it the extraction and analysis looked like more effort than it was worth, but maybe I'm wrong about that.
And for the streaming people: what actually bites you most often? State growth, skew, someone leaving a collect() in a foreachBatch? I'd like to know what to check first next time instead of starting from scratch.
Thanks,
Kiran Kumar
