Wednesday
Hi all!
Can anyone tell me if having a worker node(s) of the same type as my driver node make a difference performance-wise if I am running normal python code on a notebook as a job on this cluster? I am running mostly machine learning libraries such as torch/huggingface. Or is a single node compute cluster sufficient i.e. only a single driver/worker node? Thank you!
Kind regards,
pjv
Wednesday
Hello @pjv ,
Greetings of the day!
The choice between having a worker node of the same type as your driver node or using a single-node compute cluster depends on the nature of your workload.
Suppose you are running normal Python code on a notebook as a job on this cluster, and your workload is not distributed (i.e., it doesn't require Spark's distributed computing capabilities). In that case, a single-node compute cluster might be sufficient. In a single-node compute cluster, the driver acts as both master and worker, with no worker nodes. This is intended for jobs that use small amounts of data or non-distributed workloads such as single-node machine learning libraries.
However, if your workload involves large-scale data processing or distributed machine learning tasks, having worker nodes of the same type as your driver node could improve performance. In a multi-node compute, worker nodes run the Spark executors and other services required for proper functioning compute. All the distributed processing happens on worker nodes.
I hope this clears your doubts!
Wednesday
Hello @pjv
Greetings!
The choice between having a worker node of the same type as your driver node or using a single-node compute cluster depends on the nature of your workload.
Suppose you are running normal Python code on a notebook as a job on this cluster, and your workload is not distributed (i.e., it doesn't require Spark's distributed computing capabilities). In that case, a single-node compute cluster might be sufficient. In a single-node compute cluster, the driver acts as both master and worker, with no worker nodes. This is intended for jobs that use small amounts of data or non-distributed workloads such as single-node machine learning libraries.
However, if your workload involves large-scale data processing or distributed machine learning tasks, having worker nodes of the same type as your driver node could improve performance. In a multi-node compute, worker nodes run the Spark executors and other services required for proper functioning compute. All the distributed processing happens on worker nodes.
I hope this answers your query!
Kind regards,
Ravi
Wednesday
The choice between having a worker node of the same type as your driver node or using a single-node compute cluster depends on the nature of your workload.
Suppose you are running normal Python code on a notebook as a job on this cluster, and your workload is not distributed (i.e., it doesn't require Spark's distributed computing capabilities). In that case, a single-node compute cluster might be sufficient. In a single-node compute cluster, the driver acts as both master and worker, with no worker nodes. This is intended for jobs that use small amounts of data or non-distributed workloads such as single-node machine learning libraries.
However, if your workload involves large-scale data processing or distributed machine learning tasks, having worker nodes of the same type as your driver node could improve performance. In a multi-node compute, worker nodes run the Spark executors and other services required for proper functioning compute. All the distributed processing happens on worker nodes.
Wednesday
Hello @pjv ,
Python code will run solely on the driver node, for this case only the driver compute matters.
If you're submitting any pyspark transformations or actions through your code it will generate spark plans that will be later executed via the executors.
Thursday
Thanks @raphaelblg ! That makes sense.
Excited to expand your horizons with us? Click here to Register and begin your journey to success!
Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!