I have a custom application/executable that I upload to DBFS and transfer to my cluster's local storage for execution. I want to call multiple instances of this application in parallel, which I've only been able to successfully do with Python's subprocess.Popen(). However, doing it this way doesn't take advantage of autoscaling.

As a quick code example of what I'm trying to do:

ListOfCustomArguments = ["/path/to/config1.txt", "/path/to/config2.txt"] # Hundreds of custom configurations here
 
processes = []
for arg in ListOfCustomArguments :
   command = "/path/to/executable " + arg
   processes.append(subprocess.Popen(command, shell=True))
 
for p in processes:
   p.wait()
 
print("Done!")

As is, this will not auto-scale. Any ideas?