topic Re: Executors getting killed while Scaling Spark jobs on GPU using RAPIDS(NVIDIA) in Data Engineering

Executors getting killed while Scaling Spark jobs on GPU using RAPIDS(NVIDIA)

rajanchaturvedi — Mon, 16 Jun 2025 09:49:47 GMT

Hi Team ,

I want to take advantage of Spark Distribution over GPU clusters using RAPID(NVIDIA) , everything is setup

1. The Jar is loaded correctly via Init script , the jar is downloaded and uploaded on volume (workspace is unity enabled) and via Init script uploaded to databricks jar location

src="/Volumes/ml_apps_ml_dev/volumes/team-volume-ml_apps_nonprod/rapids-4-spark_2.12-25.04.0.jar"

DEST="/databricks/jars/rapids-4-spark_2.12-25.04.0.jar"

cluster that I am using

Spark configuration that I am using

After all this configuration I can see GPU optimizations kick in Query Execution Plan as below but when I run the spark join like join , the executors are getting killed and the spark job is stuck , kindly please help

Re: Executors getting killed while Scaling Spark jobs on GPU using RAPIDS(NVIDIA)

NandiniN — Mon, 03 Nov 2025 02:50:03 GMT

hey @rajanchaturvedi ,

Executor termination, especially when scaling a GPU-accelerated job, is almost always due to memory over-allocation (Out Of Memory, or OOM) on the worker nodes, which causes the cluster manager to kill the process. This is exacerbated in GPU environments because two large memory spaces must be managed: the CPU (JVM) heap and the dedicated GPU memory.

Can you please check and share the cause of the executors are getting killed. Also the job being stuck does not sound right? Are you certain there is nothing running?

Re: Executors getting killed while Scaling Spark jobs on GPU using RAPIDS(NVIDIA)

NandiniN — Mon, 03 Nov 2025 07:22:01 GMT

Also try to gradually reduce spark.executor.memory You need to allocate less memory to the JVM heap because the GPU needs a large chunk of the node's off-heap (system) memory. The GPU memory is allocated outside the JVM heap. If the heap is too large, it crowds out the native memory required by RAPIDS/CUDA.

Again reduce gradually

spark.rapids.memory.gpu.allocFraction from the default (usually 0.5 or 0.8). Try 0.4 or 0.3