I have a custom application/executable that I upload to DBFS and transfer to my cluster's local storage for execution. I want to call multiple instances of this application in parallel, which I've only been able to successfully do with Python's subprocess.Popen(). However, doing it this way doesn't take advantage of autoscaling.
As a quick code example of what I'm trying to do:
ListOfCustomArguments = ["/path/to/config1.txt", "/path/to/config2.txt"] # Hundreds of custom configurations here
processes = []
for arg in ListOfCustomArguments :
command = "/path/to/executable " + arg
processes.append(subprocess.Popen(command, shell=True))
for p in processes:
p.wait()
print("Done!")
As is, this will not auto-scale. Any ideas?
