Can python futures utilise all cluster nodes ?

Alex0101
New Contributor II

I used python futures to call a function multiple times concurrently, however I am not sure if all nodes is utilised or how to make sure it use all cluster nodes.

Can you confirm if I create a cluster with 5 works each with 8 memory cores for example. does that mean I can run 5 x 8 concurrent tasks ?

or the futures and python will use only the main node for each tasks ?

Code example:

# assuming 5 workers each with 8 cores
executor = ProcessPoolExecutor(5 * 8)
 
def tester():
   # code to run any parallel task
   return result
 
for index in range(10000):
   executor.submit(tester)

In other words, can the python futures or any python threading library use all cluster workers cpus ?