The potential root cause could be high GPU utilization while running a live experiment. This can be validated both by using the Spark UI and by using the Nvidia -smi command.

If a single GPU is explicitly used, this might cause an overload and hence OOM issues. 

To avoid OOM issues consider the following suggestions:

• Experiment with smaller batch sizes
• Use a larger GPU
• Use a different framework such as Horovod