NandiniN
Databricks Employee
Databricks Employee

hey @rajanchaturvedi , 

Executor termination, especially when scaling a GPU-accelerated job, is almost always due to memory over-allocation (Out Of Memory, or OOM) on the worker nodes, which causes the cluster manager to kill the process. This is exacerbated in GPU environments because two large memory spaces must be managed: the CPU (JVM) heap and the dedicated GPU memory.

Can you please check and share the cause of the executors are getting killed. Also the job being stuck does not sound right? Are you certain there is nothing running?