How to get response from API call made via executor

vaibhavaher2025
New Contributor

‎05-02-2025 06:52 AM

Hi Guys,

I'm trying to call multiple APIs via executor using foreach partition, However as API response is getting returned at executor level I'm unable to see the response of API weather its 200 or 500.

I dont want my APIs to execute on driver so I'm avoiding toPandas or Multi threading on Driver side.

Note: I've enabled logs to save in the Volume but there is no trace of the return response from API in the executor logs.

My_dataframe.repartition(10).foreachPartition(lambda partition: process_partition(partition))
logger.info(temp_dict)
 
Here I'm storing the response in temp_dict
0 Kudos

sarahbhord
Databricks Employee
Databricks Employee

‎05-07-2025 09:20 AM

Vaibhavaher2025 - 

I recommend trying the following: 

1. Write logs from executors to persist storage insideprocess_partition.

2. Use mapPartitions instead offoreachPartition to return responses back to the driver as a Dataframe

3. Check executor logs in Spark UI under Executors -> Logs for anylogger.info outputs.

Remember, foreachPartition doesn’t return data, so to see API responses, either write to external storage or use mapPartitions to collect results.

0 Kudos