mark_ott
Databricks Employee
Databricks Employee

I'm guessing you have Shuffle Write Sizes that are > 1GB.  That's when things start going down the rathole with things like Spill and OOM.  Here's a few questions I have for your.  Is Adaptive Query Execution enabled? Also I say in your earlier screen shot you had some nasty Java Garbage Collection.  Is your Cluster Photon-enabled?  This can reduce the JGC.