https://community.databricks.com/t5/support-faqs/what-is-the-best-way-to-train-a-deeplearning-model-to-ensure-we/ta-p/56899 <DIV class="lia-message-template-content-zone"> <P><SPAN>You should use distributed training.</SPAN></P> <P><SPAN>By distributing the training workload among GPUs or worker nodes, you can optimize resource utilization and reduce the likelihood of <FONT face="courier new,courier">ConnectionException</FONT> errors and out of memory (OOM) issues.</SPAN></P> <P><SPAN>A good option for distributed training is Horovod, a distributed deep learning framework.</SPAN></P> <P><SPAN>The following resources can provide guidance on how to set up and use Horovod with </SPAN><SPAN>Databricks:</SPAN></P> <UL> <LI><SPAN><STRONG>HorovodRunner: distributed deep learning with Horovod</STRONG> (<A title="HorovodRunner: distributed deep learning with Horovod (AWS)" href="https://docs.databricks.com/en/machine-learning/train-model/distributed-training/horovod-runner.html" target="_blank" rel="noopener">AWS</A> | <A title="HorovodRunner: distributed deep learning with Horovod (Azure)" href="https://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/distributed-training/horovod-runner" target="_blank" rel="noopener">Azure</A> | <A title="HorovodRunner: distributed deep learning with Horovod (GCP)" href="https://docs.gcp.databricks.com/en/machine-learning/train-model/distributed-training/horovod-runner.html" target="_blank" rel="noopener">GCP</A>)</SPAN></LI> <LI><SPAN><STRONG>Distributed training</STRONG> (<A title="Distributed training (AWS)" href="https://docs.databricks.com/en/machine-learning/train-model/distributed-training/index.html" target="_blank" rel="noopener">AWS</A> | <A title="Distributed training (Azure)" href="https://learn.microsoft.com/azure/databricks/machine-learning/train-model/distributed-training/" target="_blank" rel="noopener">Azure</A> | <A title="Distributed training (GCP)" href="https://docs.gcp.databricks.com/en/machine-learning/train-model/distributed-training/index.html" target="_blank" rel="noopener">GCP</A>)</SPAN></LI> </UL> </DIV> Thu, 11 Jan 2024 01:00:01 GMT Adam_Pavlacka 2024-01-11T01:00:01Z https://community.databricks.com/t5/support-faqs/what-is-the-best-way-to-train-a-deeplearning-model-to-ensure-we/ta-p/56899 <DIV class="lia-message-template-content-zone"> <P><SPAN>You should use distributed training.</SPAN></P> <P><SPAN>By distributing the training workload among GPUs or worker nodes, you can optimize resource utilization and reduce the likelihood of <FONT face="courier new,courier">ConnectionException</FONT> errors and out of memory (OOM) issues.</SPAN></P> <P><SPAN>A good option for distributed training is Horovod, a distributed deep learning framework.</SPAN></P> <P><SPAN>The following resources can provide guidance on how to set up and use Horovod with </SPAN><SPAN>Databricks:</SPAN></P> <UL> <LI><SPAN><STRONG>HorovodRunner: distributed deep learning with Horovod</STRONG> (<A title="HorovodRunner: distributed deep learning with Horovod (AWS)" href="https://docs.databricks.com/en/machine-learning/train-model/distributed-training/horovod-runner.html" target="_blank" rel="noopener">AWS</A> | <A title="HorovodRunner: distributed deep learning with Horovod (Azure)" href="https://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/distributed-training/horovod-runner" target="_blank" rel="noopener">Azure</A> | <A title="HorovodRunner: distributed deep learning with Horovod (GCP)" href="https://docs.gcp.databricks.com/en/machine-learning/train-model/distributed-training/horovod-runner.html" target="_blank" rel="noopener">GCP</A>)</SPAN></LI> <LI><SPAN><STRONG>Distributed training</STRONG> (<A title="Distributed training (AWS)" href="https://docs.databricks.com/en/machine-learning/train-model/distributed-training/index.html" target="_blank" rel="noopener">AWS</A> | <A title="Distributed training (Azure)" href="https://learn.microsoft.com/azure/databricks/machine-learning/train-model/distributed-training/" target="_blank" rel="noopener">Azure</A> | <A title="Distributed training (GCP)" href="https://docs.gcp.databricks.com/en/machine-learning/train-model/distributed-training/index.html" target="_blank" rel="noopener">GCP</A>)</SPAN></LI> </UL> </DIV> Thu, 11 Jan 2024 01:00:01 GMT https://community.databricks.com/t5/support-faqs/what-is-the-best-way-to-train-a-deeplearning-model-to-ensure-we/ta-p/56899 Adam_Pavlacka 2024-01-11T01:00:01Z