How can I resolve CUDA OOM issues while performing batch inference in notebooks using ML models?

Adam_Pavlacka — Thu, 11 Jan 2024 01:00:00 GMT

The potential root cause could be high GPU utilization while running a live experiment. This can be validated both by using the Spark UI and by using the Nvidia -smi command.

If a single GPU is explicitly used, this might cause an overload and hence OOM issues.

To avoid OOM issues consider the following suggestions:

Experiment with smaller batch sizes
Use a larger GPU
Use a different framework such as Horovod

article How can I resolve CUDA OOM issues while performing batch inference in notebooks using ML models? in Support FAQs

How can I resolve CUDA OOM issues while performing batch inference in notebooks using ML models?