topic Re: Can python futures utilise all cluster nodes ? in Data Engineering

Can python futures utilise all cluster nodes ?

Alex0101 — Wed, 29 Jun 2022 09:51:47 GMT

I used python futures to call a function multiple times concurrently, however I am not sure if all nodes is utilised or how to make sure it use all cluster nodes.

Can you confirm if I create a cluster with 5 works each with 8 memory cores for example. does that mean I can run 5 x 8 concurrent tasks ?

or the futures and python will use only the main node for each tasks ?

Code example:

# assuming 5 workers each with 8 cores
executor = ProcessPoolExecutor(5 * 8)
 
def tester():
   # code to run any parallel task
   return result
 
for index in range(10000):
   executor.submit(tester)

In other words, can the python futures or any python threading library use all cluster workers cpus ?

Re: Can python futures utilise all cluster nodes ?

-werners- — Wed, 29 Jun 2022 10:36:57 GMT

it will run on the main node (driver) only.

You need some kind of cluster management framework to distribute the work over node, like Yarn, Spark Dask, Ray etc

If you would use pyspark, then you can leverage the parallel processing of Spark and it would indeed run over multiple nodes, if your function uses Spark.

Re: Can python futures utilise all cluster nodes ?

Keyuri — Wed, 29 Jun 2022 16:44:17 GMT

You can create a init script and then add it during cluster start up

Re: Can python futures utilise all cluster nodes ?

Alex0101 — Wed, 29 Jun 2022 17:52:22 GMT

Can you elaborate, which init script to add to the cluster ?