I have a function making api calls. I want to run this function in parallel so I can use the workers in databricks clusters to run it in parallel. I have tried

with ThreadPoolExecutor() as executor:

 results = executor.map(getspeeddata, alist)

to run my function but this does not make use of the workers and runs everything on the driver. How do I make my function run in parallel?