topic How can I use cluster autoscaling with intensive subprocess calls? in Data Engineering

How can I use cluster autoscaling with intensive subprocess calls?

KellenO — Thu, 08 Dec 2022 20:30:22 GMT

I have a custom application/executable that I upload to DBFS and transfer to my cluster's local storage for execution. I want to call multiple instances of this application in parallel, which I've only been able to successfully do with Python's subprocess.Popen(). However, doing it this way doesn't take advantage of autoscaling.

As a quick code example of what I'm trying to do:

ListOfCustomArguments = ["/path/to/config1.txt", "/path/to/config2.txt"] # Hundreds of custom configurations here
 
processes = []
for arg in ListOfCustomArguments :
   command = "/path/to/executable " + arg
   processes.append(subprocess.Popen(command, shell=True))
 
for p in processes:
   p.wait()
 
print("Done!")

As is, this will not auto-scale. Any ideas?

Re: How can I use cluster autoscaling with intensive subprocess calls?

Anonymous — Fri, 09 Dec 2022 00:18:17 GMT

Autoscaling works for spark jobs only. It works by monitoring the job queue, which python code won't go into. If it's just python code, try single node.

https://docs.databricks.com/clusters/configure.html#cluster-size-and-autoscaling

Re: How can I use cluster autoscaling with intensive subprocess calls?

tunstila — Fri, 09 Dec 2022 14:23:13 GMT

Nice response @Joseph Kambourakis