https://community.databricks.com/t5/machine-learning/outofmemoryerror-cuda-out-of-memory-on-llm-finetuning/m-p/61255#M3040 <DIV class=""><SPAN class=""><SPAN>I am trying to finetune <STRONG>llama2_lora</STRONG> model using the <STRONG>xTuring</STRONG> library, while facing this error. (batch size is 1).&nbsp;</SPAN>I am working on a cluster having&nbsp;<STRONG>1 Worker (</STRONG></SPAN><SPAN class=""><STRONG>28&nbsp;GB Memory,&nbsp;</STRONG><SPAN><STRONG>4&nbsp;Cores)</STRONG> and&nbsp;</SPAN></SPAN><STRONG><SPAN class="">1 Driver (</SPAN><SPAN class="">110&nbsp;GB Memory,&nbsp;16&nbsp;Cores).</SPAN></STRONG></DIV><DIV class="">&nbsp;</DIV><DIV class=""><SPAN class="">I am facing this error:&nbsp;<SPAN class="">OutOfMemoryError: </SPAN><SPAN>CUDA out of memory. Tried to allocate 86.00 MiB (GPU 0; 15.57 GiB total capacity; 8.02 GiB already allocated; 57.44 MiB free; 8.02 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF.</SPAN><BR /><BR />It says that the&nbsp;<SPAN>total capacity is 15.57 GBs. Does this memory represents any of the worker or driver memory? If yes, should it be more than 15.57? Is the current implementation not able to utilize the available memory?<BR /></SPAN></SPAN></DIV> Tue, 20 Feb 2024 12:40:24 GMT hv129 2024-02-20T12:40:24Z https://community.databricks.com/t5/machine-learning/outofmemoryerror-cuda-out-of-memory-on-llm-finetuning/m-p/61255#M3040 <DIV class=""><SPAN class=""><SPAN>I am trying to finetune <STRONG>llama2_lora</STRONG> model using the <STRONG>xTuring</STRONG> library, while facing this error. (batch size is 1).&nbsp;</SPAN>I am working on a cluster having&nbsp;<STRONG>1 Worker (</STRONG></SPAN><SPAN class=""><STRONG>28&nbsp;GB Memory,&nbsp;</STRONG><SPAN><STRONG>4&nbsp;Cores)</STRONG> and&nbsp;</SPAN></SPAN><STRONG><SPAN class="">1 Driver (</SPAN><SPAN class="">110&nbsp;GB Memory,&nbsp;16&nbsp;Cores).</SPAN></STRONG></DIV><DIV class="">&nbsp;</DIV><DIV class=""><SPAN class="">I am facing this error:&nbsp;<SPAN class="">OutOfMemoryError: </SPAN><SPAN>CUDA out of memory. Tried to allocate 86.00 MiB (GPU 0; 15.57 GiB total capacity; 8.02 GiB already allocated; 57.44 MiB free; 8.02 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF.</SPAN><BR /><BR />It says that the&nbsp;<SPAN>total capacity is 15.57 GBs. Does this memory represents any of the worker or driver memory? If yes, should it be more than 15.57? Is the current implementation not able to utilize the available memory?<BR /></SPAN></SPAN></DIV> Tue, 20 Feb 2024 12:40:24 GMT https://community.databricks.com/t5/machine-learning/outofmemoryerror-cuda-out-of-memory-on-llm-finetuning/m-p/61255#M3040 hv129 2024-02-20T12:40:24Z