Databricks Community

HamzaJosh · 10-27-2021

I have a function making api calls. I want to run this function in parallel so I can use the workers in databricks clusters to run it in parallel. I have tried with ThreadPoolExecutor() as executor: results = executor.map(getspeeddata, alist)to run m...

HamzaJosh · 11-01-2021

You guys are not getting the point, I am making API calls in a function and want to store the results in a dataframe. I want multiple processes to run this task in parallel. How do I create a UDF and use it in a dataframe when the task is calling an ...

HamzaJosh · 10-28-2021

Thanks Hubert and werners for responding. Please give me some urls which shows how I can create UDF's. Do i still need to use threadpool? How do I make it run in parallel after using a UDF? I am newbie and need more than just create a UDF. Please he...

Databricks Community

User Stats

User Activity

I want to use databricks workers to run a function in parallel on the worker nodes

Re: I want to use databricks workers to run a function in parallel on the worker nodes

Re: I want to use databricks workers to run a function in parallel on the worker nodes