Asynchronous API calls from Databricks Workflow jo...

pjv · ‎05-10-2024

Hi all,

I have many API calls to run on a python Databricks notebook which I then run regularly on a Databricks Workflow job. When I test the following code on an all purpose cluster locally i.e. not via a job, it runs perfectly fine. However, when I run the same notebook on a job it does not work anymore. The calls are run sequentially instead of in parallel. Does anyone know why and what I can do to fix it?

Thank you!

Here is my code:

import asyncio

import requests

import nest_asyncio

nest_asyncio.apply()

async def with_threads():

def make_request(😞

response = requests.get('https://www.google.com')

return response

reqs = [asyncio.to_thread(make_request) for _ in range(0,20)]

responses = await asyncio.gather(*reqs)

return responses

async_result = asyncio.run(with_threads())

PS: The request and loop is different in my original code and only used here to explain the problem.

Asynchronous API calls from Databricks Workflow job