https://community.databricks.com/t5/technical-blog/ml-training-tip-of-the-week-1-optimizing-gpu-utilization-in/ba-p/86677 <P><SPAN>When running distributed training or batch inference on multi-node GPU clusters with Spark, the GPUs on the Driver node often remain underutilized, resulting in unnecessary waste of GPU resources. The figures below illustrate this issue:</SPAN>&nbsp;<span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Fig.1: Only one GPU in Driver node is being utilized" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/10828i180382AE1C7151A4/image-size/large?v=v2&amp;px=999" role="button" title="driver_GPUutil.png" alt="Fig.1: Only one GPU in Driver node is being utilized" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Fig.1: Only one GPU in Driver node is being utilized</span></span></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Fig.2: All GPUs in the Worker node are utilized." style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/10827iA88CCE78DB028B25/image-size/large?v=v2&amp;px=999" role="button" title="worker_GPUutil.png" alt="Fig.2: All GPUs in the Worker node are utilized." /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Fig.2: All GPUs in the Worker node are utilized.</span></span></P> <P><STRONG style="font-family: inherit;">Solution: Heterogeneous Compute Types for Driver and Worker Nodes</STRONG></P> <P><SPAN>To address this problem, you can select different compute types for the Driver and Worker nodes. For example, you might choose a CPU instance for the Driver node and a GPU instance for the Worker nodes.</SPAN></P> <P><SPAN>Currently, the cluster UI does not support heterogeneous compute types. However, you can create such a cluster using the following API command:</SPAN></P> <P>&nbsp;</P> <LI-CODE lang="python">clusters create --json '{ "cluster_name": "xxxx", "spark_version": "14.3.x-gpu-ml-scala2.12", "node_type_id": "g4dn.12xlarge", "driver_node_type_id": "i3.xlarge", "autoscale" : { "min_workers": 1, "max_workers": 2 }, "aws_attributes" : {"first_on_demand": 3} }' </LI-CODE> <P>&nbsp;</P> <P>&nbsp;</P> <P>#automl #modeltraining #mosaicai</P> Thu, 05 Sep 2024 19:56:33 GMT Lanz 2024-09-05T19:56:33Z https://community.databricks.com/t5/technical-blog/ml-training-tip-of-the-week-1-optimizing-gpu-utilization-in/ba-p/86677 <P>Welcome to the&nbsp;<EM>ML Training Tip Of The Week </EM>Series</P> Thu, 05 Sep 2024 19:56:33 GMT https://community.databricks.com/t5/technical-blog/ml-training-tip-of-the-week-1-optimizing-gpu-utilization-in/ba-p/86677 Lanz 2024-09-05T19:56:33Z