I'm running into an issue where my Spark driver keeps pausing and eventually restarting due to excessive garbage collection (GC), even though I’ve already scaled up the cluster memory. Below is an example from the driver logs:

Driver/192.168.231.23 paused the JVM process 74 seconds during the past 120 seconds (61.71%) because of GC. We observed 3 such issue(s) since 2025-06-04T07:34:56.504Z.

Attached is a screenshot showing multiple gc_pressure warnings over the past hour, with pauses consistently exceeding 60 seconds out of every 120.

So far I’ve:

• Increased driver memory.

• Monitored memory usage and confirmed it’s not fully consumed, but GC still takes a long time.

My questions:

1. What are the most common root causes for this kind of sustained GC pressure in Spark drivers?

2. Are there any specific Spark configurations or GC tuning parameters that could help mitigate this?

3. Would refactoring code to reduce memory usage on the driver (e.g. avoiding collect(), broadcasting, etc.) be more effective?

4. Is there a recommended way to identify large object allocations or memory leaks that lead to GC overload?

Any insight or guidance would be greatly appreciated!

Thanks!