Driver and worker node utalisation

[pjv]

Hi all! 

Can anyone tell me if having a worker node(s) of the same type as my driver node make a difference performance-wise if I am running normal python code on a notebook as a job on this cluster? I am running mostly machine learning libraries such as torch/huggingface. Or is a single node compute cluster sufficient i.e. only a single driver/worker node? Thank you!

Kind regards,

pjv

[Ravivarma]

Hello @pjv ,

Greetings of the day!

The choice between having a worker node of the same type as your driver node or using a single-node compute cluster depends on the nature of your workload.

Suppose you are running normal Python code on a notebook as a job on this cluster, and your workload is not distributed (i.e., it doesn't require Spark's distributed computing capabilities). In that case, a single-node compute cluster might be sufficient. In a single-node compute cluster, the driver acts as both master and worker, with no worker nodes. This is intended for jobs that use small amounts of data or non-distributed workloads such as single-node machine learning libraries.
However, if your workload involves large-scale data processing or distributed machine learning tasks, having worker nodes of the same type as your driver node could improve performance. In a multi-node compute, worker nodes run the Spark executors and other services required for proper functioning compute. All the distributed processing happens on worker nodes.

I hope this clears your doubts!

[Ravivarma]

Hello @pjv 

Greetings!

The choice between having a worker node of the same type as your driver node or using a single-node compute cluster depends on the nature of your workload.

Suppose you are running normal Python code on a notebook as a job on this cluster, and your workload is not distributed (i.e., it doesn't require Spark's distributed computing capabilities). In that case, a single-node compute cluster might be sufficient. In a single-node compute cluster, the driver acts as both master and worker, with no worker nodes. This is intended for jobs that use small amounts of data or non-distributed workloads such as single-node machine learning libraries.

However, if your workload involves large-scale data processing or distributed machine learning tasks, having worker nodes of the same type as your driver node could improve performance. In a multi-node compute, worker nodes run the Spark executors and other services required for proper functioning compute. All the distributed processing happens on worker nodes.

I hope this answers your query!

Kind regards,

Ravi

[Ravivarma]

The choice between having a worker node of the same type as your driver node or using a single-node compute cluster depends on the nature of your workload.

Suppose you are running normal Python code on a notebook as a job on this cluster, and your workload is not distributed (i.e., it doesn't require Spark's distributed computing capabilities). In that case, a single-node compute cluster might be sufficient. In a single-node compute cluster, the driver acts as both master and worker, with no worker nodes. This is intended for jobs that use small amounts of data or non-distributed workloads such as single-node machine learning libraries.

However, if your workload involves large-scale data processing or distributed machine learning tasks, having worker nodes of the same type as your driver node could improve performance. In a multi-node compute, worker nodes run the Spark executors and other services required for proper functioning compute. All the distributed processing happens on worker nodes.

[raphaelblg]

Hello @pjv , 

Python code will run solely on the driver node, for this case only the driver compute matters. 

If you're submitting any pyspark transformations or actions through your code it will generate spark plans that will be later executed via the executors. 

[pjv]

Thanks @raphaelblg ! That makes sense.