https://community.databricks.com/t5/get-started-discussions/driver-and-worker-node-utalisation/m-p/75883#M7691 <P>Hello&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/105077">@pjv</a>&nbsp;</P> <P>Greetings!</P> <P>The choice between having a worker node of the same type as your driver node or using a single-node compute cluster depends on the nature of your workload.</P> <P>Suppose you are running normal Python code on a notebook as a job on this cluster, and your workload is not distributed (i.e., it doesn't require Spark's distributed computing capabilities). In that case, a single-node compute cluster might be sufficient. In a single-node compute cluster, the driver acts as both master and worker, with no worker nodes. This is intended for jobs that use small amounts of data or non-distributed workloads such as single-node machine learning libraries.</P> <P>However, if your workload involves large-scale data processing or distributed machine learning tasks, having worker nodes of the same type as your driver node could improve performance. In a multi-node compute, worker nodes run the Spark executors and other services required for proper functioning compute. All the distributed processing happens on worker nodes.</P> <P>I hope this answers your query!</P> <P>Kind regards,</P> <P>Ravi</P> Wed, 26 Jun 2024 19:03:37 GMT Ravivarma 2024-06-26T19:03:37Z https://community.databricks.com/t5/get-started-discussions/driver-and-worker-node-utalisation/m-p/75855#M7689 <P>Hi all!&nbsp;</P><P>Can anyone tell me if having a worker node(s) of the same type as my driver node make a difference performance-wise if I am running normal python code on a notebook as a job on this cluster? I am running mostly machine learning libraries such as torch/huggingface. Or is a single node compute cluster sufficient i.e. only a single driver/worker node? Thank you!</P><P>Kind regards,</P><P>pjv</P> Wed, 26 Jun 2024 13:53:31 GMT https://community.databricks.com/t5/get-started-discussions/driver-and-worker-node-utalisation/m-p/75855#M7689 pjv 2024-06-26T13:53:31Z https://community.databricks.com/t5/get-started-discussions/driver-and-worker-node-utalisation/m-p/75880#M7690 <P>Hello&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/105077">@pjv</a>&nbsp;,</P> <P>Greetings of the day!</P> <P>The choice between having a worker node of the same type as your driver node or using a single-node compute cluster depends on the nature of your workload.</P> <P>Suppose you are running normal Python code on a notebook as a job on this cluster, and your workload is not distributed (i.e., it doesn't require Spark's distributed computing capabilities). In that case, a single-node compute cluster might be sufficient. In a single-node compute cluster, the driver acts as both master and worker, with no worker nodes. This is intended for jobs that use small amounts of data or non-distributed workloads such as single-node machine learning libraries.<BR />However, if your workload involves large-scale data processing or distributed machine learning tasks, having worker nodes of the same type as your driver node could improve performance. In a multi-node compute, worker nodes run the Spark executors and other services required for proper functioning compute. All the distributed processing happens on worker nodes.</P> <P>I hope this clears your doubts!</P> Wed, 26 Jun 2024 18:59:26 GMT https://community.databricks.com/t5/get-started-discussions/driver-and-worker-node-utalisation/m-p/75880#M7690 Ravivarma 2024-06-26T18:59:26Z https://community.databricks.com/t5/get-started-discussions/driver-and-worker-node-utalisation/m-p/75883#M7691 <P>Hello&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/105077">@pjv</a>&nbsp;</P> <P>Greetings!</P> <P>The choice between having a worker node of the same type as your driver node or using a single-node compute cluster depends on the nature of your workload.</P> <P>Suppose you are running normal Python code on a notebook as a job on this cluster, and your workload is not distributed (i.e., it doesn't require Spark's distributed computing capabilities). In that case, a single-node compute cluster might be sufficient. In a single-node compute cluster, the driver acts as both master and worker, with no worker nodes. This is intended for jobs that use small amounts of data or non-distributed workloads such as single-node machine learning libraries.</P> <P>However, if your workload involves large-scale data processing or distributed machine learning tasks, having worker nodes of the same type as your driver node could improve performance. In a multi-node compute, worker nodes run the Spark executors and other services required for proper functioning compute. All the distributed processing happens on worker nodes.</P> <P>I hope this answers your query!</P> <P>Kind regards,</P> <P>Ravi</P> Wed, 26 Jun 2024 19:03:37 GMT https://community.databricks.com/t5/get-started-discussions/driver-and-worker-node-utalisation/m-p/75883#M7691 Ravivarma 2024-06-26T19:03:37Z https://community.databricks.com/t5/get-started-discussions/driver-and-worker-node-utalisation/m-p/75884#M7692 <P>The choice between having a worker node of the same type as your driver node or using a single-node compute cluster depends on the nature of your workload.</P> <P>Suppose you are running normal Python code on a notebook as a job on this cluster, and your workload is not distributed (i.e., it doesn't require Spark's distributed computing capabilities). In that case, a single-node compute cluster might be sufficient. In a single-node compute cluster, the driver acts as both master and worker, with no worker nodes. This is intended for jobs that use small amounts of data or non-distributed workloads such as single-node machine learning libraries.</P> <P>However, if your workload involves large-scale data processing or distributed machine learning tasks, having worker nodes of the same type as your driver node could improve performance. In a multi-node compute, worker nodes run the Spark executors and other services required for proper functioning compute. All the distributed processing happens on worker nodes.</P> Wed, 26 Jun 2024 19:05:37 GMT https://community.databricks.com/t5/get-started-discussions/driver-and-worker-node-utalisation/m-p/75884#M7692 Ravivarma 2024-06-26T19:05:37Z https://community.databricks.com/t5/get-started-discussions/driver-and-worker-node-utalisation/m-p/75889#M7693 <P>Hello&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/105077">@pjv</a>&nbsp;,&nbsp;</P> <P>Python code will run solely on the driver node, for this case only the driver compute matters.&nbsp;</P> <P>If you're submitting any pyspark transformations or actions through your code it will generate spark plans that will be later executed via the executors.&nbsp;</P> Wed, 26 Jun 2024 19:55:19 GMT https://community.databricks.com/t5/get-started-discussions/driver-and-worker-node-utalisation/m-p/75889#M7693 raphaelblg 2024-06-26T19:55:19Z https://community.databricks.com/t5/get-started-discussions/driver-and-worker-node-utalisation/m-p/75922#M7694 <P>Thanks&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/97998">@raphaelblg</a>&nbsp;! That makes sense.&nbsp;</P> Thu, 27 Jun 2024 07:05:14 GMT https://community.databricks.com/t5/get-started-discussions/driver-and-worker-node-utalisation/m-p/75922#M7694 pjv 2024-06-27T07:05:14Z https://community.databricks.com/t5/get-started-discussions/driver-and-worker-node-utalisation/m-p/78049#M7695 <P>Yes, you are right, I agree with you.</P> Wed, 10 Jul 2024 07:58:48 GMT https://community.databricks.com/t5/get-started-discussions/driver-and-worker-node-utalisation/m-p/78049#M7695 SeanRay 2024-07-10T07:58:48Z