topic Re: CUDA out of memory in Machine Learning

CUDA out of memory

gary7135 — Thu, 20 Jul 2023 16:28:09 GMT

I am trying out the new Meta LLama2 model.

Following the databricks provided notebook example: https://github.com/databricks/databricks-ml-examples/blob/master/llm-models/llamav2/llamav2-13b/01_load_inference.py

I keep getting CUDA out of memory. My GPU cluster runtime is

13.2 ML (includes Apache Spark 3.4.0, GPU, Scala 2.12), with 256GB memory and 1 GPU

Error message:

CUDA out of memory. Tried to allocate 314.00 MiB (GPU 0; 14.76 GiB total capacity; 13.50 GiB already allocated; 313.75 MiB free; 13.51 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

What would be a good way to solve this issue?

Re: CUDA out of memory

Kumaran — Fri, 21 Jul 2023 14:41:46 GMT

Hi @gary7135,

Thank you for posting the question in the Databricks community.

Kindly update the configuration by setting fp16=True instead of its current value of false. For further information regarding the CUDA error related to this, please refer to this documentation

Re: CUDA out of memory

gary7135 — Fri, 21 Jul 2023 17:54:20 GMT

Thank you. Can you provide example of how to set this argument in notebooks?

Re: CUDA out of memory

Kumaran — Fri, 21 Jul 2023 18:32:57 GMT

Hello @gary7135,

Thank you for the response.

According to GitHub (you shared above), you should have a configuration file where you need to make this settings. Please refer to the image below for more details:

Re: CUDA out of memory

gary7135 — Fri, 21 Jul 2023 20:25:55 GMT

Thank you. I am running this python file directly in Databricks notebook https://github.com/databricks/databricks-ml-examples/blob/master/llm-models/llamav2/llamav2-7b/01_load_inference.py

The file does not seem to reference the config json file?

Re: CUDA out of memory

Anonymous — Sun, 23 Jul 2023 04:43:07 GMT

Hi @Kumaran

Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.

Cheers!

Re: CUDA out of memory

Kumaran — Mon, 24 Jul 2023 07:44:21 GMT

Hi @gary7135,

If you are now following the same as GitHub.You may somehow work on how to point out the configuration fp16=True to your file,