How to get response from API call made via executor

vaibhavaher2025 — Fri, 02 May 2025 13:52:41 GMT

Hi Guys,

I'm trying to call multiple APIs via executor using foreach partition, However as API response is getting returned at executor level I'm unable to see the response of API weather its 200 or 500.

I dont want my APIs to execute on driver so I'm avoiding toPandas or Multi threading on Driver side.

Note: I've enabled logs to save in the Volume but there is no trace of the return response from API in the executor logs.

My_dataframe.repartition(10).foreachPartition(lambda partition: process_partition(partition))

logger.info(temp_dict)

Here I'm storing the response in temp_dict

Re: How to get response from API call made via executor

sarahbhord — Wed, 07 May 2025 16:20:51 GMT

Vaibhavaher2025 -

I recommend trying the following:

1. Write logs from executors to persist storage insideprocess_partition.

2. Use mapPartitions instead offoreachPartition to return responses back to the driver as a Dataframe

3. Check executor logs in Spark UI under Executors -> Logs for anylogger.info outputs.

Remember, foreachPartition doesn’t return data, so to see API responses, either write to external storage or use mapPartitions to collect results.

topic How to get response from API call made via executor in Data Engineering

How to get response from API call made via executor

Re: How to get response from API call made via executor